Introduction

In this file we will go through the annotations and clouds of 13 adjectives: heilzaam, hoekig, gekleurd, dof, hachelijk, geestig, hoopvol, hemels, geldig, gemeen, goedkoop, grijs, heet: a more general description of their selection can be found here. For each of them, the sense distribution, a sort of confusion matrix and a description of the clouds will be shown. The descriptions can still go much deeper; for now the priority is an overview of the possibilities and variation across lemmas before going too deep into each of them. At the same time, they are still only descriptions and no conclusions are being drawn from them yet, so beyond this introduction the rest is not summarized yet.

When examples are cited, the target item is highlighted in bold and color; context words taken by a model (by default, the one with least restrictions) may be emphasized in italics and those selected by annotators as cues may be boldened, depending on what need to be illustrated. Some of the cues may be wrongly annotated: there was a bug in the annotation tool by which, upon registering context words, a word with the same wordform as the actual goal but in a previous position might be tagged in its stead. This was noticed during the annotation process and the annotators (especially those who had already sent their work) were warned, but some might not have checked their results properly. I will clean it, eventually (they are quite evident).

Sense distribution

The different adjectives exemplify different polysemy phenomena: metaphor, metonymy, similarity and more complex mixes. The sense tags have codes described in a table with definitions at the beginning of each section, but the annotators also had the option of assigning a geen ‘none of the above’ tag, in which case they had to add an explanatory comment.

When setting up the annotation procedure, pilot batches of 40-50 concordance lines of each type were collected to estimate the frequency of the senses we expected (we did have to exclude candidates because some sense was not frequent enough). The annotation of the pilot sets was not extremely thorough and sometimes we modified the definition set afterwards, so it’s best to keep that in mind when reading the comparison between the expected distribution and what came out of the annotations. It’s worthy to consider that if my estimates over samples of 40-50 tokens match what we find in a bigger sample (and especially if it’s robust across batches) it’s quite encouraging. If (the skewness of) the sense distribution turns out to be a factor in the topology of the clouds, it’s useful to know that it can be estimated from such a small sample.

The Sense distribution subsection of each section compares then the estimated sense distribution based on the pilot concordances with the one found in each batch and the whole set of tokens. For each type a plot is shown with a row of dots per batch over a line and two more rows below the line representing the pilot-based estimate and the overall distribution. Circle dots represent tokens tagged with a given sense by the majority of the annotators (each sense has a color) and triangles represent either tokens primarily annotated with the geen ‘none of the above’ tag or tokens for which the annotators did not agree at all. The dots in the batch rows represent one token each and their transparency codes the mean confidence after standardizing its value by annotator and lemma.

At the end of the section a small summary will be made of the structure of each lemma: it is important to keep in mind that this is highly dependent on the corpus and their language.

Confusion matrix

For each type, two confusion matrices will be shown in the Confusion matrix subsection. In each of them, a row represents a majority sense (or no_agreement if there was no majority) and a column represents a sense tag. By hovering over the row names it’s possible to retrieve the definition, since the tags are not precisely transparent. Since the geen ‘none of the above’ tags can have different reasons, the annotators’ explanations were classified with the following tags:

  • between, when the annotator reported doubt between two or more of the given senses;
  • not_listed, when the explanations referred to a sense that was not contemplated in the list of senses (or not understood as such);
  • unclear, when the explanations referred to either insufficient or unclear context (or simply difficulty to understand, such as “geen flauw idee”), and
  • wrong_lemma, when they referred to an issue with lemmatization, part-of-speech tagging (including parts of proper nouns) or even spelling, so that the target didn’t actually correspond to what was meant to be annotated.

The first matrix shows raw annotation counts. Each cell tells the number of tokens with the majority sense of the row that were tagged with the sense of the column: The cell in the row of heilzaam_1 and the column heilzaam_2 will say how many tokens with the majority sense heilzaam_1 received some heilzaam_2 annotation. The totals indicate the number of tokens that were tagged with a given sense (for column totals). The first descriptions only focus on which senses are confused with each other. The caption also records the proportion of tokens with a certain majority sense that received the same tag from all annotators.

The second (“weighted”) matrix shows the mean of the mean confidences of the annotations. Suppose the row is heilzaam_1 and the column is heilzaam_2; to fill in such a cell, for each token with majority sense heilzaam_1 the mean of the confidences of the heilzaam_2 annotations is computed. Since the heilzaam_2 is not the majority sense, there won’t be more than one annotation of the same token to average across; for the heilzaam_1 column, each token would have two to four agreeing annotations, and their respective confidences would then be averaged to reach one mean confidence per token per sense. The final value of the cel is the mean, across all tokens of that cell, of those mean confidences.

Confidence values range from 0 to 5 but were represented to the annotators as a star rating: there was no option to color no stars, so one star is “minimum confidence” (0 in numbers) and the full set is “maximum confidence” (5 in numbers). In some descriptions, confidences between 0 and 1 will be considered “low”, between 2 and 3, “medium”, and 4 or 5 “high”, but in practice the great mejority of the confidence ratings is high. That also means that when looking at the mean confidences, while 2.5 could be considered a medium value, in practice it is quite low value, below the normal mean. Therefore, values of the weighted matrix that are equal to or greater than the mean confidence values of the whole type will be darker and boldened, against lighter values that are lower. This number will also be reported in the table caption, along with the median.

Nephology

For each type, the Nephology subsection discusses the role of the parameters in the structure of the cloud of clouds (level 1 of the visualization) showing at least two color coded plots, and then compare sets of models (level 2). First, parameters that seem to have little to no effect in the variation between models are kept constant to compare the resulting selection; then other combinations that might provide different results are explored, and finally some combination of parameters that seems to provide “satisfying” models is kept constant look at the actual effect of the less important parameters. Normally, the strongest ones are those that select first order context features, while the second order parameters rarely make much of a difference.

The comparison between models normally takes the following steps:

  1. examine the range of distances between the models through the distance matrix;
    This can be illustrated with a distance matrix like the one available in the visualization and/or with a plot of distances by parameter: each dot is a pair of models that only differ across one parameter, their position in the plot is given by the parameter that varies (x-axis) and the distance between the two models (y-axis).
  2. describe the general look of the clouds without color coding and how they change between MDS and t-SNE solutions;
  3. color code with sense tags and describe the revealed structure.
    This can be illustrated with sets of clouds of some configuration, like what would be seen in Level 2 of the visualization (but not interactive)

The description includes the behaviour of outliers in MDS solutions, the separability of senses in any kind of solutions, how many and how clear clusters show up in the different t-SNE solutions (if there is any that provides a particularly good representation, how robust it is across different perplexities) and how such structure relates to the parameters under comparison. For now, individual tokens are not examined but on an exceptional basis. A certain bias should be acknowledge: certain settings tend to be preferred (from theoretical reasons sometimes, but not always) and the findings in one type definitely affect how the following ones are understood. Hopefully, time and experience will provide the tools to revise these decisions with better criteria.

At the end of the subsection a next course of action is suggested, such as promising model(s) and which tokens seem interesting to look at. That is highlighted in a nice quote block at the end of each section.

To ease the descriptions, the parameters will be written in all caps and their values will follow them separated by colon. These are:

First order part-of-speech (FOC-POS)
Can take the value FOC-POS:nav, when only nouns, adjectives and verbs were selected as first order features, or FOC-POS:all, when there was no such restriction (still, some part-of-speech tags were ignored always, such as interjections).
The tendency is to default to FOC-POS:nav, since function words are probably less informative (kind of linguistically informed default).
First order window (FOC-WIN)
Can take the value FOC-WIN:5, when only features within a 5-5 window of the target were included, or FOC-WIN:10, when a 10-10 window was used.
The tendency is to default to FOC-WIN:10, to allow for more information; normally relying on other restrictions is enough to filter out the noise.
Positive pointwise mutual information as filter (PPMI)
Can take the value PPMI:weight when the second order vectors are weighted by the PPMI value between the first order feature they represent and the target type, PPMI:selection when only features with a positive PMI with the target type were included but the vectors were not weighted, and PPMI:no when no such filter was applied. Normally, the models with PPMI:selection are more similar to PPMI:no than to PPMI:weight and they are not considered in the initial comparisons.
The initial tendency was to default to PPMI:no, since a high PPMI value signals a feature as characteristic of the type rather than of groups of it (like a sense), but in the analyses described in this file it never performs as well as the alternatives.
Vector length (LENGTH)
Can take the values LENGTH:5000 and LENGTH:10000 when the 5000/10000 most frequent features were used as second order dimensions, or LENGTH:FOC when the same first order dimensions are used for the second order. That means that their number and frequency depends on the result of the first order restrictions for that particular sample of tokens. Normally, while this is not an extremely strong parameter, LENGTH:FOC can make a different against the other two, frequency based values.
The tendency is default to LENGTH:FOC because it should be better tailored to the specific context of the tokens in the cloud; it’s harder to compare clouds with different first order context words, but it does seem to perform better in most cases. Between frequency based values, I almost never look at LENGTH:10000, since it almost never seems to make much of a difference, but I probably should look into it before discarding it from future clouds. If both frequency based settings perform very similarly, smaller numbers should be preferred (hence the tendency to choose LENGTH:FOC as well, since it normally means fewer than 5000 dimensions).
Second order part-of-speech (SOC-POS)
Can take the values SOC-POS:nav or SOC-POS:all and refers to a filter on the second order dimensions. This never makes much of a difference.
The tendency is to default to SOC-POS:nav (De Pascale, 2019, pp. 62–63).
Second order window (SOC-WIN)
Can take the values SOC-WIN:4 or SOC-WIN:10 depending on whether the PPMI values for the second order vectors were computed based on a 4-4 or 10-10 window.
This parameter seems to group models for some types, but doesn’t really affect the structure of the clouds that much as far as I can see. The tendency is to default to SOC-WIN:4 (See De Pascale, 2019, pp. 62–63). Could it be that in the cases where it seems relevant, what actually happens is that all the other parameters are just too weak?

Eventually, it would be nice to reinstate sentence boundary as parameter (replacing for example SOC-POS): currently, this parameter is fixed to only count context words within sentence boundaries. The difference between LENGTH:5000 and LENGTH:10000 also seems neglectable.

heilzaam

The adjective heilzaam was tagged with 2 definitions, reproduced in Table 1: the most frequent one refers literally to the domain of health (heilzaam_1) and the other one to a broader application of “beneficial” (heilzaam_2). There is expectation of a relatively high number of adverbial uses.

Table 1. Definitions of ‘heilzaam’.
code definition example freq
heilzaam_1 (letterlijk) bijdragend tot gezondheid en lichamelijk welzijn een heilzaam dieet 22
heilzaam_2 (figuurlijk) nuttig, een gunstig effect hebbend een heilzaam besluit 12

Sense distribution

The sample consists of 240 tokens (6 batches) out of 1476 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 1. The distributions of the annotations (not majority senses) by annotator are shown in Figure 2. Batch 5 was annotated by 4 annotators. Against what was expected from the distribution in the pilot set, the metaphorical heilzaam_2 ‘beneficial’ was more frequent in most batches and the overall distribution than the literal heilzaam_2 ‘healthy’. There were also almost no geen ‘none of the above’ cases. It’s left for manual inspection whether that is really the case in these concordances or the annotators just did not exclude adverbial uses, given that the same sense tags could be applied to those.

Figure 1. Distribution of majority senses of 'heilzaam' per batch

Figure 1. Distribution of majority senses of ‘heilzaam’ per batch

Figure 2. Distribution of sense annotations of 'heilzaam' per annotator, grouped by batch.

Figure 2. Distribution of sense annotations of ‘heilzaam’ per annotator, grouped by batch.

“heilzaam” is an adjective with two senses of similar frequencies, a more frequent metaphorical one with application to a number of domains, and a literal, more restricted one.

Confusion matrix

The confusion matrix between the majority senses and other tagged senses can be seen in Table 2 (raw number of tokens with such senses assigned) and Table 3 (mean confidence of such sense annotation in each token). Given that the literal heilzaam_1 ‘healthy/healing’ sense is the core sense and more specific, we could expect more confidence and agreement for that sense and rare cases where heilzaam_2 ‘beneficial’ would be applied to tokens with majority heilzaam_1 ‘healing’ sense. However, the table shows that there is a higher proportion of heilzaam_1 ‘healing’ tokens with heilzaam_2 ‘beneficial’ as alternative than the other way around, which could suggest an interpretation of the relationship between senses as specialization (heilzaam_1 ‘healing’ as a specific case of heilzaam_2 ‘beneficial’). There are also three tokens with no agreement between the annotators.

Table 2. Non weighted sense matrix of ‘heilzaam’ senses. Proportion of tokens with full agreement per sense-tag is: heilzaam_1: 0.66, heilzaam_2: 0.81.
senses heilzaam_1 heilzaam_2 between not_listed unclear
heilzaam_1 98 29 1 1 2
heilzaam_2 25 139 1 1 0
no_agreement 3 3 1 0 0
total 126 171 3 2 2
Table 3. Weighted sense matrix of ‘heilzaam’ senses. Mean confidence across the lemma is 4; values above are darker and boldened. Median confidence across the lemma is 4.
senses heilzaam_1 heilzaam_2 between not_listed unclear
heilzaam_1 4.03 3.24 0 0 2
heilzaam_2 3.72 4.02 2 2 0
no_agreement 3.33 3.33 2 0 0

The tokens with no agreement are transcribed in (1) through (3). The first two were annotated by four annotators, half of which chose one or the other sense tag; (3) instead was assigned heilzaam_1 ‘healing’ by one annotator, heilzaam_2 ‘beneficial’ by another, and geen ‘none of the above’ by the third, with the comment “Ik twijfel tussen beide opties.” (“I hesitate between both options”).

  1. Come As You Are ’ , maar in zijn laatste interviews sprak hij vaak over het heilzame gebruik van kogels afvuren in het bos en poseerde hij enkele weken voor zijn dood ’
    As You Are ’ , but in his last interviews he talked about the beneficial use of shooting bullets in the forest and he posed some weeks before his death
  2. bij de groep en zanger Karl Hyde worstelde met zijn alcoholverslaving . Maar de heilzame bezinning die volgde , leverde een herboren Underworld en een bijzonder fris klinkende nieuwe langspeler op
    was held in the group and the singer Karl Hyde struggled with his alcoholism. But the healing/beneficial reflection that followed produced a reborn Underworld and a new longplayer that sounded particularly fresh
  3. . Hippotherapie - - het omgaan met pony’s en paarden - - kan erg heilzaam zijn voor gehandicapte kinderen . " Het zou erg zijn als educatieve projecten die
    disabled people. Hippotherapy - - hanging out with pony’s and horses - - can be really healthy/beneficial for handicapped children. It would be great if educational projects that

Nephology of heilzaam

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of heilzaam created on 24/03/2020, modeling between 235 and 239 tokens. The stress value of the MDS solution for the cloud of models is 0.138.

The main distinction between the models is made by FOC:WIN across the y-axis; within each hemisphere there are three clear groups, one for PPMI:weight, cut orthogonally by FOC-POS and SOC-WIN , and one for each FOC-POS, cut orthogonally by PPMI (excluding PPMI:weight) and SOC-WIN (see Figure 3, Figure 4, Figure 5 ). Based on the distance matrices, where a change in only SOC-WIN makes a difference between 0.04 and 0.05 for PPMI:weight and between 0.06 and 0.10 otherwise, it would seem that the clustering power of SOC-WIN is visibile because of the weakness of the other parameters rather than its own strength. Figure 6 shows the range of distances between models that only vary along one parameter.

Figure 3. Cloud of models of 'heilzaam'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=heilzaam'> here</a>.

Figure 3. Cloud of models of ‘heilzaam’. Explore it here.

Figure 6. Distances between models of 'heilzaam' that vary along only one parameter, colored by `PPMI`.

Figure 6. Distances between models of ‘heilzaam’ that vary along only one parameter, colored by PPMI.

To compare the effects of the strongest parameters, we’ll first look at models with PPMI:weight | PPMI:no + LENGTH:FOC + SOC-POS:nav + SOC-WIN:4. Few tokens seem to be lost by stricter parameters: 1 by FOC-POS:nav, 2 for FOC-WIN:5 combined with one more restriction and 4 with all three restrictions. The distance matrix (Distance matrix 1) confirms the observation from the cloud of models that PPMI:weight models are more similar to each other than PPMI:no models and also shows that the least restrictive model is the most different, with distances of at least 0.35 and normally above 0.5 to other models.

Distance matrix 1. Distance matrix between some models of ‘heilzaam’
id FOC-POS FOC-WIN PPMI
1 nav 10_10 weight
2 nav 10_10 no
3 all 10_10 weight
4 all 10_10 no
5 nav 5_5 weight
6 nav 5_5 no
7 all 5_5 weight
8 all 5_5 no

The MDS solutions with PPMI:weight look more evenly spread than their PPMI:no counterparts, which are more compact with some satellites; the loosest model seems more sensitive to some outliers, the farthest of which is lost by FOC-POS:nav models and lies in the periphery of any other model (example (4)). The loosest PPMI:selection looks even worse, with another powerful outlier (example (5)).1 With color coding all models show a relatively good split. The effect of the PPMI parameter is even more evident in the t-SNE solutions: from perplexity 5, the PPMI:no models already look like a rather uniform mass without clusters that only gets worse with higher perplexity, except maybe in the FOC-WIN:5 + FOC-POS:nav model. The PPMI:weight models still have an archipelago configuration with perplexity 5, although it’s possible to distinguish two main areas even without color coding; these areas become more distinct with higher perplexity, although the improvement between 30 and 50 doesn’t seem big. The PPMI:weight + FOC-WIN:5 models seem to have the best separability.

  1. Viagra heilzaam tegen oogaandoeningen NEW YORK - Dat Viagra uitstekende diensten kan bewijzen als
    Viagra beneficial (healthy) against eye conditions NEW YORK - That Viagra can perform exceptional services
  2. ijzer dan een grote biefstuk . Bevat enzymen die spijsvertering bevorderen . Heilzaam bij menstruele problemen of tijdens menopauze . Twee eetlepels leveren 20 % van de
    then a big steak. Includes enzymes that induce digestion. Beneficial (healthy) for menstrual problems or during menopause. Two teaspoons provide 20% of

With regard to the color coding (Figure 7), the heilzaam_1 ‘healing’ tokens do seem to group together, but as the two main clouds become more distinct, it also becomes clear that they group with tokens of heilzaam_2 ‘beneficial’, always the same bunch: these are most of the heilzame werking ‘beneficial effect’ cases. There is also a group of heilzaam_1 ‘healing’ tokens bridging the two clouds in the FOC-WIN:10 + PPMI:weight models, which, curiously enough, forms a separate group from the rest of heilzaam_1 ‘healing’ in the FOC-WIN:10 + FOC-POS:nav + PPMI:no model with perplexity 20 (they seem to have mostly werking, but also frequent prepositions). Finally, a small tight cluster can be seen in the PPMI:weight models, with tokens that also stick together, albeit less tightly, in all other models: those are joined by the occurence of effect ‘effect’.

By keeping the stronger parameters constant, the other parameters make very little difference, with maximum values around 0.20 in their respective distance matrices.

Figure 7. Tokens of 'heilzaam' in the t-SNE solutions (perplexity 30) of the selected models

Figure 7. Tokens of ‘heilzaam’ in the t-SNE solutions (perplexity 30) of the selected models

For further inspection of “heilzaam” clouds it might be interesting to select PPMI:weight + SOC-POS:nav + SOC-WIN:4 + LENGTH:FOC models.

hoekig

The adjective hoekig was tagged with 3 definitions, reproduced in Table 4: the most frequent sense refers to literally angulous objects (hoekig_1), then an abstract metaphoric extension refers to broken movements and melodies (hoekig_2) and the least frequent to clumsy people (hoekig_3).

Table 4. Definitions of ‘hoekig’.
code definition example freq
hoekig_1 (van voorwerpen, figuren e.d.) met hoeken of scherpe kanten een hoekige tekening, een hoekig gezicht 19
hoekig_2 (van bewegingen, ritmes e.d.) niet vloeiend een hoekig melodietje 11
hoekig_3 (van personen) houterig, stijf, onhandig in de omgang een hoekig karakter 5

Sense distribution

The sample consists of 280 tokens (7 batches) out of 1242 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 8. The distributions of the annotations (not majority senses) by annotator are shown in Figure 9. No batch was annotated by 4 annotators. From the pilot set, a number of ambiguous or adverbial cases was expected, but that did not emerge from the annotation. In any case, the sense proportions are overall similar to the estimate, although with hoekig_2 ‘broken-movement’ almost as frequent as hoekig_1 ‘angulous’, just as frequent in some of the batches.

Figure 8. Distribution of majority senses of 'hoekig' per batch

Figure 8. Distribution of majority senses of ‘hoekig’ per batch

Figure 9. Distribution of sense annotations of 'hoekig' per annotator, grouped by batch.

Figure 9. Distribution of sense annotations of ‘hoekig’ per annotator, grouped by batch.

“hoekig” is an adjective with three main senses, a concrete and an abstract one of similar frequencies and one less frequent and exclusively applied to human beings.

Confusion matrix

Matrices

The confusion matrix between the majority senses and other tagged senses can be seen in Table 5 (raw number of tokens with such senses assigned) and Table 6 (mean confidence of such sense annotation in each token). The different senses of this lemma are characterized by the kind of nouns the adjective is predicated of, so there shouldn’t be much confusion between the annotations, unless the concordances themselves are too vague. The number of disagreeing annotations is indeed rather low, but actually surprisingly high for hoekig_3 ‘clumsy’, which in fact has a proportion of full agreement of only 0.5.

There are six cases with no agreement between annotators, three of which were the exact same expression in some sort of cultural agenda, as shown in (6): the left context before Hengst was different each time. The three tokens were annotated by the same three annotators, and all three had the same reaction (probably corrected) across all tokens: two assigned the geen ‘none of the above’ tag, one explaining that they didn’t understand the word bokstragiek and both at some point suggesting it might be part of a title; the other one consistently assigned hoekig_2 ‘broken-movement’ with confidence 4 of 5 (while the others assigned 2 and 0). The other examples with no agreement will be transcribed in the “Confusing examples” subsection.

  1. Hengst Hoekige bokstragiek in een solo van John Buysman , met de blanke blues als begeleiding
    Stallion Broken ??tragedy in a solo from John Buysman, with the blank blues leading
Table 5. Non weighted sense matrix of ‘hoekig’ senses. Proportion of tokens with full agreement per sense-tag is: hoekig_1: 0.88, hoekig_2: 0.76, hoekig_3: 0.5.
senses hoekig_1 hoekig_2 hoekig_3 between not_listed unclear wrong_lemma
hoekig_1 138 5 8 2 1 1 0
hoekig_2 6 100 18 0 0 0 0
hoekig_3 3 10 34 1 1 2 0
not_listed 0 1 0 0 1 0 0
unclear 0 0 1 0 0 1 0
no_agreement 3 4 1 2 1 4 3
total 150 120 62 5 4 8 3

According to the weighted confidence matrix, the confidence of the agreeing annotations of hoekig_1 ‘angulous’ and hoekig_2 ‘broken-movement’ are relatively high, but those of hoekig_3 ‘clumsy’ are rather low (although higher than half the points), while annotations of hoekig_3 ‘clumsy’ for tokens with hoekig_2 ‘broken-movement’ have a higher mean confidence.

Table 6. Weighted sense matrix of ‘hoekig’ senses. Mean confidence across the lemma is 4.05; values above are darker and boldened. Median confidence across the lemma is 4.
senses hoekig_1 hoekig_2 hoekig_3 between not_listed unclear wrong_lemma
hoekig_1 4.24 4 3.75 0 0 0 0
hoekig_2 2.67 4.09 4.22 0 0 0 0
hoekig_3 1.67 3.5 3.81 3 3 2.5 0
not_listed 0 0 0 0 3.5 0 0
unclear 0 0 2 0 0 2 0
no_agreement 3.33 3.5 2 1 3 1.5 0.67

Confusing examples

This adjective has provided on one hand a number of cases with no agreement on one hand and on the other unexpected cases of confusion, such as instances tagged as hoekig_3 ‘clumsy’ by the majority but as something else by a disagreeing annotator.

In examples (7) through (9) the annotators couldn’t agree, but for different reasons. For (7), one annotator assigned hoekig_1 ‘angulous’ with confidence 2 and added that they doubted between hoekig_1 ‘angulous’ in a figurative sense and hoekig_3 ‘clumsy’ but favoured the first one; the other two assigned geen ‘none of the above’ instead, but with very different explanations. One said they were doubting between hoekig_2 ‘broken-movement’ and hoekig_3 ‘clumsy’ and the other, that the right sense was hoekig_1 ‘angulous’ but in metaphorical sense2, thus actually agreeing with the first annotator.

  1. het er anders aan toe : hij zet de zaak graag op scherp met dwarse , hoekige en heel persoonlijke lezingen . Hij laat je als luisteraar geen moment met rust
    […] he sharpened the topic with crossed, angulous and very personal lessons. Hi didn’t give you as listener any break and

Example (8) also received one hoekig_1 ‘angulous’ tag (this time with confidence of 4) and two geen ‘none of the above’ annotations, one of which declared hesitation between hoekig_1 ‘angulous’ and hoekig_2 ‘broken-movement’ and the other reported lack of context. In this case, the whole context could be understood either in terms of a plastic or graphic artwork, or in terms of music: the first annotator must have felt primed towards the first case (the cue they annotated is: vormen (R0)), while the others reflected on the broader spectrum of interpretations (the cues they tagged where compositie (L5), ritme (L8), vormen (R0) and none respectively.

  1. de voorgrond . In het ritme van de compositie en de spanning van de hoekige vormen trachtten ze tevens het hectische stadsleven weer te geven . De stijleenheid was
    in the forground. In the rhythm of the composition and the tension of the angulous forms they also attempted to illustrate the hectic city life. The style unit was rather

Example (9) instead received one annotation of each of the available senses, with confidence of 4 for the hoekig_1 ‘angulous’ tag and of 2 for the rest.

  1. de verte , een keigroen grasveld omzoomd door een zacht gebold , spierwit auditorium , een hoekige blik op een knetterblauwe hemel , een patio van stoffig zand , een groepje berkenbomen in
    green, a very green field of grass fringed by a slightly curved, bright white auditorium, an angulous look at a crackling blue sky, a patio of dusty sand, a small group of beech trees

The number of disagreeing hoekig_3 ‘clumsy’ cases is rather high; a first exploration showed a variety of complicated nouns modified by hoekig (interview, lichaamstaal ‘body language’, denken ‘thoughts’) and some adverbial uses, but it would be necessary to check the other concordances to see how rare that really is.

Nephology of hoekig

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of hoekig created on 24/03/2020, modeling between 275 and 278 tokens.The stress value of the MDS solution for the cloud of models is 0.157.

The main division in the cloud of models is made by FOC-WIN along the vertical axis, forming two parallel diagonal stripes. Each main cloud has three clear subclouds, one on the left edge for FOC-POS:nav, organized by PPMI and SOC-WIN in orthogonal dimensions, one in the middle for FOC-POS:all + PPMI:weight, further split by SOC-WIN, and a more disperse one on the right with the rest, also organiwed by PPMI and SOC-WIN in orthogonal dimensions (see Figure 10.

Figure 10. Cloud of models of 'hoekig'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=hoekig'> here</a>.

Figure 10. Cloud of models of ‘hoekig’. Explore it here.

According to the distance matrices, SOC-WIN doesn’t really make much difference, with distances between 0.05 and 0.07 when PPMI:weight (which does tend to have small differences between models) and 0.05 and 0.11 otherwise (where the difference made by other parameters is much greater). LENGTH tends to make bigger differences, in the range of 0.1 and 0.15 (see Figure 11. Therefore, to compare the effect of the stronger parameters, the following selection will be used: PPMI:weight | PPMI:no + LENGTH:FOC + SOC-WIN:4 + SOC-POS:nav.

Figure 11. Distances between models of 'hoekig' varying along only one parameter, colored by `PPMI`.

Figure 11. Distances between models of ‘hoekig’ varying along only one parameter, colored by PPMI.

Few tokens are lost by restrictions on the selection of first order context words: 1 with FOC-POS or PPMI filters, 2 with FOC-WIN combined with FOC-POS and 3 with FOC-WIN combined with PPMI. The distance matrix (Distance matrix 2) show that the FOC-WIN:5 models are the most similar to each other and that the FOC-POS:all + PPMI:no (particularly if also FOC-WIN:10) are the most different from the rest.

Distance matrix 2. Distance matrix between some models of ‘hoekig’
id FOC-POS FOC-WIN PPMI
1 nav 10_10 weight
2 nav 10_10 no
3 all 10_10 weight
4 all 10_10 no
5 nav 5_5 weight
6 nav 5_5 no
7 all 5_5 weight
8 all 5_5 no

The MDS solutions don’t look too different from each other: the PPMI:no models look more compact, with a clear outlier in the loosest model that is lost by all the FOC-POS:nav | PPMI:weight models (10) and an even more clear outlier in FOC-WIN:5 + FOC-POS:nav + PPMI:no model that is not actually remarkable in other models (example (11), with the context words from such model in italics).

  1. met trompet en piano ingeduffelde ’ Lotus ’ , het technoslaapliedje ’ Asleep ’ of het hoekige ’ Many Weathers Apart ’ . Rangschikken onder : eclectisch , ambitieus en intrigerend
    ‘Lotus’ muffled with trumpet and piano, the techno lullaby ‘Asleep’ or the broken/angulous ‘Many Weathers Apart’. Classified under: eclectic, ambitious and intriguing
  2. geweten van de dienstreizen van Van der Stoel , zoals kroonprins Willem-Alexander ( 33 ) de hoekige Minister van Staat ( 76 ) eenvoudig modern noemt . Uit het optreden van
    known of the business trips of Van der Stoel, like the crown prince Willem-Alexander (33) calls the clumsy Minister of State (76) simply modern. From the performance of

Finally, by keeping FOC-WIN:10 + FOC-POS:all + PPMI:weight + LENGTH:FOC | LENGTH:5000, the distance matrix has values between 0.01 and 0.17 and the SOC-POS:nav + SOC-WIN:4 + LENGTH:FOC still looks like one of the best options, but SOC-POS:all + LENGTH:5000 | LENGTH:FOC could have better separability.

For further inspection of “hoekig” it would be interesting to delve deeper into SOC-WIN:4 + FOC-WIN:10 + FOC-POS:all + PPMI:weight.

gekleurd

The adjective gekleurd was tagged with 3 definitions, reproduced in Table 7. The most frequent means literally colored (gekleurd_1) and two less frequent senses are metaphorical extensions, one to refer to POC, people of color (gekleurd_2), and the other one to biased or corrupted discourse and actions (gekleurd_3). The expected number of ambiguous/adverbial cases is auite small.

Table 7. Definitions of ‘gekleurd’.
code definition example freq
gekleurd_1 met kleur, in letterlijke zin (in het bijzonder, niet zwart, wit of grijs) gekleurde wangen 26
gekleurd_2 (van personen e.a.) niet blank de gekleurde medemens, van gekleurde afkomst zijn 7
gekleurd_3 (van uitspraken, opvattingen e.d.) niet neutraal, tendentieus een gekleurde voorstelling van zaken 5

Sense distribution

The sample consists of 280 tokens (7 batches) out of 4520 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 12. The distributions of the annotations (not majority senses) by annotator are shown in Figure 13. Batch 1 was annotated by 4 annotators. There is some variation across batches, but the overall distribution is very similar to the expected one, with some more cases of gekleurd_3 ‘tainted’ and none of geen ‘none of the above’.

Figure 12. Distribution of majority senses of 'gekleurd' per batch

Figure 12. Distribution of majority senses of ‘gekleurd’ per batch

Figure 13. Distribution of sense annotations of 'gekleurd' per annotator, grouped by batch.

Figure 13. Distribution of sense annotations of ‘gekleurd’ per annotator, grouped by batch.

“gekleurd” is an adjective with three senses, one literal very frequent and two less frequent metaphorical extensions.

Confusion matrix

Matrices

The confusion matrix between the majority senses and other tagged senses can be seen in Table 8 (raw number of tokens with such senses assigned) and Table 9 (mean confidence of such sense annotation in each token). The expected confusion between the senses is quite low, although there could be some cases where the adjective applied to a person could be more ambiguous, especially if it is itself modified by an adverb. The matrix shows indeed very few cases of confusion between the senses and high confidence values in the agreeing annotations, but also rather high confidence values in some of the disagreeing ones. Examples of cases with no agreement and not_listed as majority sense are described in their respective subsections.

Table 8. Non weighted sense matrix of ‘gekleurd’ senses. Proportion of tokens with full agreement per sense-tag is: gekleurd_1: 0.94, gekleurd_2: 0.76, gekleurd_3: 0.67.
senses gekleurd_1 gekleurd_2 gekleurd_3 between not_listed unclear
gekleurd_1 182 2 6 0 4 1
gekleurd_2 3 42 6 0 0 1
gekleurd_3 6 1 48 0 7 3
not_listed 0 1 1 0 2 1
no_agreement 6 2 4 2 2 0
total 197 48 65 2 15 6
Table 9. Weighted sense matrix of ‘gekleurd’ senses. Mean confidence across the lemma is 4.53; values above are darker and boldened. Median confidence across the lemma is 5.
senses gekleurd_1 gekleurd_2 gekleurd_3 between not_listed unclear
gekleurd_1 4.71 2 4 0 2 0
gekleurd_2 4.67 4.69 3.5 0 0 2
gekleurd_3 3.67 4 4.29 0 3.57 1.67
not_listed 0 4 5 0 3 0
no_agreement 2.67 1.75 2.88 1.5 2.5 0

No agreement

The examples (12) through (17) resulted in no agreement between the annotators. In their texts, the context words tagged as cues by them are boldened, with in superscript the number of annotators that selected them.

The first two were annotated by 4 annotators that split in two different opinions: (12) between gekleurd_1 ‘colored’ (with confidences of 0 and 5, the latter commenting it was metaphorical) and gekleurd_3 ‘tainted’ (with confidences of 4 and 5), (13) between gekleurd_1 ‘colored’ (confidences of 2 and 5) and gekleurd_2 ‘POC’ (confidences of 2 and 3).

  1. we Toneelgroep De Appel weer : als de producent van vitaal en circusachtig toneel1 , vrolijk4 gekleurd maar met1 een1 toefje1 verdriet2 . T/m 25 nov Den Haag Appeltheater .
    us Theater group De Appel: as the producer of vital and circus-like theater, cheerfully colored but with a touch of sadness. Until Nov 25 The Hague Appeltheater.
  2. elkaar aan . Aller aandacht is gericht op dat ene exemplaar1 , een groengeel3 gekleurd vrouwtje4 , dat kennelijk pas uit1 het overwinteringsgebied3 ( zuidelijk1 Afrika1 ) is aangekomen .
    to each other. All the attention is directed to that one individual, a greenish yellow colored little woman, who apparently had just arrived from the hibernation area (Southern Africa).

The annotators of (14) were split between gekleurd_1 ‘colored’, gekleurd_3 ‘tainted’ and not_listed (with a comment suggesting it might be gekleurd_3 ‘tainted’ or a fixed expression). For all three cases the confidence was low (1 or 2).

  1. kriebelen . Het gaat daarbij vooral om die warme , door een roze2 bril2 gekleurde herinneringen2 . Natuurlijk weten ze wel dat deze vaak niets meer te
    itch. It’s mainly about those warm memories colored by pink glasses. Obviously they know that these often don’t [have] anything to [do]

The disagreement in the annotation of (15) was between gekleurd_1 ‘colored’, gekleurd_2 ‘POC’ (with a comment that it could be gekleurd_1 ‘colored’) and a hesitation between gekleurd_1 ‘colored’ and gekleurd_3 ‘tainted’ but minimum confidence. The other two annotations had confidence of 2 and 1 respectively.

  1. uit vroeger tijden . Zoals ijzervlechter Thomas Paepcke , die zijn gespierde , gezond1 gekleurde lichaam2 weinig rust gunt . De Duitser is nog slechts gekleed in een afgescheurde
    from earlier times. Like the steel fixer Thomas Paepcke, who gives little rest to his muscular, healthily colored body. The German is still barely dressed in a torn off

For the last two cases the disagreement was between gekleurd_1 ‘colored’, gekleurd_3 ‘tainted’ and geen ‘none of the above). With (16), that geen was a between the other two suggestions, while for (17) it was a not_listed with the suggestion “aangepast” ’adapted’, and the annotator that chose gekleurd_1 ‘colored’ added it was a figurative meaning.

  1. in de hoogbeschaafde landen . De televisie2 die vierentwintig uur op vierentwintig uur haar gekleurde beelden2 in de huiskamers straalt1 . Is dat nu nodig ? Maar
    in the civilized countries. The television that 24 hours a day radiates colored images in the rooms fo the house. Is that really necessary? But
  2. De Koreaan Hye-Soo Sonn was over de hele lijn vocaal degelijk , zij het te kunstmatig2 gekleurd , maar had weinig te vertellen . Zelfs zijn vaak geprezen catalogusaria uit Mozarts
    The Korean Hye-Soo Sonn’s performance was in general vocally decent, although too artificially colored, but had little to tell. Even his often praised catalog aria from Mozart

Not listed

For the examples (18) and (19) the majority of the annotators assigned the geen ‘none of the above’ tag and commented that the right sense was not listed among the options. In their texts, the context words tagged as cues by them are boldened, with in superscript the number of annotators that selected them.

Example (18) was annotated by four annotators, two of which assigned geen ‘none of the above’ with suggestions of an unlisted sense (the comments pointed either to the sense of “divers” ‘diverse’ or to the “political context”), one assigned gekleurd_2 ‘POC’ and the other gekleurd_3 ‘tainted’, all with confidences of 4 or 5 (except for the one of “political context”). The annotations of (19), on the other hand, have relatively low confidence values; two suggest the definition of “versierd” ‘decorated’ and the other one just declares that there is not enough context.

  1. in een1 nieuw regeerakkoord1 " , schrijft Zalm . " Ik wil liever2 een gekleurde coalitie3 dan2 een2 brede2 . Zorg bij de formatie ook voor nieuwe2 gezichten2 van1
    in a new coalition agreement“, writes Zalm.”I prefer a colored coalition than a broad one. Take care when forming it to also include new faces of
  2. bleek het oer-Kantsjeliaanse mineur1 aan te houden dat zijn hele latere oeuvre1 kenmerkt . Gekleurd met1 dissonanten3 , aangezet met oorverdovende tutti-klappen , dan weer zachtjes treurend tot in het banale
    seems to keep the primeval Kantsjelian?? miner that characterizes all his later pieces. Colored with dissonants, initiated with deafening tutti-claps??, then again softly mourning until […] in the banal

Nephology of gekleurd

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of gekleurd created on 24/03/2020, modeling between 271 and 279 tokens.The stress value of the MDS solution for the cloud of models is 0.183.

The main distinction between the models is made by FOC-WIN, along the vertical dimension. Within each hemisphere, the groupings are a result of a combination of FOC-POS and PPMI, with FOC-POS:nav and FOC-POS:all + PPMI:weight on the left, and the rest of FOC-POS:all on the right, split by PPMI and SOC-WIN on orthogonal dimensions (Figure 14).

Figure 14. Clouds of models of 'gekleurd'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=gekleurd'> here</a>.

Figure 14. Clouds of models of ‘gekleurd’. Explore it here.

In fact, the only parameters that ever make a difference greater than 0.2 by themselves are FOC-POS and FOC-WIN as long as PPMI:no | PPMI:selection and PPMI (Figure 153) Therefore, to compare the effect of the stronger parameters, the following selection will be used: PPMI:weight | PPMI:no + LENGTH:no + SOC-WIN:4 + SOC-POS:nav.

Figure 15. Distances between models of 'gekleurd' varying along only one parameter, colored by `PPMI`.

Figure 15. Distances between models of ‘gekleurd’ varying along only one parameter, colored by PPMI.

Between 1 and 8 tokens are lost by different combinations of restrictions on the first order context words, FOC-WIN:5 being the strictest. The distance matrix has relatively low values, but the loosest model stands out with the highest of all, often above 0.5 (if PPMI:no is replaced by PPMI:selection that is still the case, but the values are a bit lower, only once above 0.5).

With an MDS solution, the models don’t look too different: the loosest looks more compact, with some satellites; the strictest has a compact half and a more disperse ones, and the rest are something in between. Color coding shows that the compact half is made almost exclusively of (the predominant) gekleurd_1 ‘colored’ tokens, while the more disperse section has two smaller groups, one for each other sense.

With t-SNE solutions, perplexity 5 only gives an archipelago, and from perplexity 20 onwards it’s possible, albeit not so evident, to distinguish two bigger groups in FOC-POS:nav | PPMI:weight models and a small tighter lump with PPMI:weight. The shapes stay in FOC-POS:nav models with perplexity 30, but otherwise the clouds look rather uniform. With perplexity 50, the PPMI:no clouds just look a bit more compact than their PPMI:weight counterparts. Color coding shows that, already from perplexity 5, thes senses tend to stick together. From perplexity 20 it would seem that there are two main groups of gekleurd_2 ‘POC’, and gekleurd_3 ‘tainted’ is a bit more clear when PPMI:weight. The small lump (of gekleurd_1 ‘colored’) seems quite constant across models, but the pattern that pulls those tokens together is not clear yet. With perplexity 30 it wold seem that FOC-POS:nav offers the best separability; PPMI:weight works better than PPMI:no, but including PPMI:selection instead gives an alternative that still seems to work well enough (Figure 16). With perplexity 50, most of the structure is lost.

Figure 16. Tokens of 'gekleurd' in the t-SNE solutions (perplexity 30) of the selected models

Figure 16. Tokens of ‘gekleurd’ in the t-SNE solutions (perplexity 30) of the selected models

For further inspection of “gekleurd” it would be interesting to look into FOC-POS:nav + PPMI:weight | PPMI:selection + LENGTH:FOC + SOC-WIN:4 + SOC-POS:nav…

dof

The adjective dof was tagged with 4 definitions, reproduced in Table 10. It basically means “dull”, and the identified senses are literal, visual (dof_1), synesthetic to sounds (dof_2), rather anthropocentric, referring to the energy of people and their emotions (dof_3) and abstract in relation to intellectual entities like memories (dof_4), not clear to the mind. The first two are expected to be equally frequent and the last two much less so; a relatively high number of unclear or adverbial cases is expected.

Table 10. Definitions of ‘dof’.
code definition example freq
dof_1 (van kleuren en zichtbare dingen) mat, zonder glans, vaal een doffe blik 12
dof_2 (van geluiden) niet luid of scherp, onderdrukt, gesmoord een doffe kreet 12
dof_3 (van personen, gevoelens e.d.) niet opgewekt, lusteloos, zonder energie doffe onverschilligheid, doffe ellende 8
dof_4 (van denkbeelden e.d.) niet scherp voor de geest staand een doffe herinnering 2

Sense distribution

The sample consists of 320 tokens (8 batches) out of 1268 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 17. The distributions of the annotations (not majority senses) by annotator are shown in Figure 18. Batch 7 was annotated by 4 annotators. The overall distribution is very similar to the estimated one, if the geen ‘none of the above’ cases from the pilot concordance are counted as dof_3 ‘energy’. While the proportions vary slightly across batches, the overall distribution seems to be balanced between the first three senses, with very few cases of dof_4 ‘dull-mental’ and with no agreement.

Figure 17. Distribution of majority senses of 'dof' per batch

Figure 17. Distribution of majority senses of ‘dof’ per batch

Figure 18. Distribution of sense annotations of 'dof' per annotator, grouped by batch.

Figure 18. Distribution of sense annotations of ‘dof’ per annotator, grouped by batch.

“dof” is an adjective with three equally frequent senses, one literal and two metaphorical, of which one is synesthetic, and a less frequent abstract sense.

Confusion matrix

Matrices

The confusion matrix between the majority senses and other tagged senses can be seen in Table 11 (raw number of tokens with such senses assigned) and Table 12 (mean confidence of such sense annotation in each token). There should not be much confusion between the senses, except maybe between the two abstract senses (dof_3 and dof_4).

Table 11. Non weighted sense matrix of ‘dof’ senses. Proportion of tokens with full agreement per sense-tag is: dof_1: 0.76, dof_2: 0.91, dof_3: 0.69, dof_4: 0.25.
senses dof_1 dof_2 dof_3 dof_4 not_listed unclear
dof_1 91 2 14 4 2 0
dof_2 1 121 8 1 1 0
dof_3 9 9 95 9 2 1
dof_4 1 0 2 4 0 0
not_listed 0 0 1 0 1 0
unclear 1 0 1 0 1 2
no_agreement 4 1 6 2 1 1
total 107 133 127 20 8 4
Table 12. Weighted sense matrix of ‘dof’ senses. Mean confidence across the lemma is 4.25; values above are darker and boldened. Median confidence across the lemma is 5.
senses dof_1 dof_2 dof_3 dof_4 not_listed unclear
dof_1 4.23 3.5 4.07 2.25 3.5 0
dof_2 4 4.61 2.62 2 2 0
dof_3 3.33 3.33 4.1 2.56 3.5 5
dof_4 3 0 5 2.88 0 0
not_listed 0 0 1 0 3.5 0
unclear 1 0 5 0 2 2
no_agreement 3.75 2 2.83 2.5 4 2

The confusion matrix shows that, indeed, there is little confusion in general, with the clear exception of dof_3 ‘energy’: as a majority sense it has a high number of alternative annotations from all the other senses (there are even more dof_4 ‘dull-mental’ annotations in tokens with dof_3 ‘energy’ as majority sense than in tokens with dof_4 ‘dull-mental’ as majority sense), and it’s a frequent alternative sense for all the other majority senses. Moreover, the alternative dof_3 ‘energy’ annotations when dof_1 ‘dull-visual’ is majority sense have a high confidence: these are mostly cases of “ogen” ‘eyes’ (7 out of 14) or “blik” ‘gaze’, and the rest are mostly visible entities with an emotional connotation in their dullness, like in (20). The cases with dof_2 ‘dull-sound’ as majority sense and dof_3 ‘energy’ as alternative are either adverbial cases like the ones reported in “Adverbial uses” or sounds with an emotional connotation in their dullness, like in (21).

  1. kenden van de zeldzame zeer oude mens . De huid van deze dieren werd dof . Zij verloren hun eetlust en zij bewogen zich nauwelijks . Tenslotte
    knew from the rare very old man. The skin of this animals became dull. They lost their appetite and barely moved. Finally
  2. op de nieuwe standplaats gemanoevreerd . Pas als de stacaravan de plaats met een doffe zucht heeft ingenomen , klaren de gezichten van de vrouwen op : dat is weer goed
    maneuvered into the new place. Only when the caravan has taken its place with a dull sigh, do the faces of the women lighten up: it is good again

The cases of confusion with dof_3 ‘energy’ as majority sense show a wide range of phenomena. Those with dof_4 ‘dull-mental’ as alternative are all abstract cases that fit better with dof_3 ‘energy’ but not so bad, mostly with nouns such as berusting ‘resignation’ (three times), its synonym gelatenheid, leegte ‘emptiness’ and herhaling ‘repetition’; the same is the case for the two concordances with dof_4 ‘dull-mental’ as majority sense and dof_3 ‘energy’ as alternative. Some of those that received dof_2 ‘dull-sound’ as an alternative were actually adverbial uses applied to sounds, but there are also some interesting challenging cases next to the more neglectable ones. These are discussed in “Interesting cases” along with most of the cases that received a dof_1 ‘dull-visual’ tag. The cases with no agreement or adverbial uses are discussed in their respective subsections. In the examples reproduced in those subsections, the context words chosen by the annotators are boldened, with in superscript the number of annotator that voted each of them.

No agreement

With examples (22) through (27) the annotators couldn’t agree on a sense. In the first three examples, there were four annotators that split in two equal groups: between dof_2 ‘dull-sound’ and dof_3 ‘energy’ in the case of (22) and between dof_1 ‘dull-visual’ and dof_3 ‘energy’ in the other two cases (the four annotators were always the same but grouped differently each time). In the first case, the dof_3 ‘energy’ annotations received maximum confidence and the dof_2 ‘dull-sound’ ones only 2, but one of the annotators of the latter group added an explanation, justifying their choice by referring to the pronunciation of “de doffe e” ‘the dull e’. In the other two cases, the dof_1 ‘dull-visual’ annotations always had higher confidence than their dof_3 ‘energy’ counterparts, but it must also be noted that the target is actually an adverb in both examples, applied to the verbs kijken ‘look’ and afsteken ‘contrast’.

  1. Luna1 klinkt vooreerst mooi . Het woord1 baadt niet in die onbeklemtoonde1 , doffe sfeer4 van de Nederlandse1 taal1 met zijn Peters1 en Veerles1 . Namen op een
    Luna sounds first of all beautiful. The word does not bathe in the unstressed, dull sphere of the Dutch language with its Peters and Veerles. Names in a
  2. " Met dank aan de regen " MELBOURNE – Marat1 Safin1 keek4 dof uit2 zijn2 ogen2 . Hij had net in de derde set een 6-3 in
    “Thanks to the rain” MELBOURNE – Marat Safin looked dull from his eyes. He had just in the third set […] a 6-3
  3. permitteren , en waarbij de1 pakken1 van gouverneur1 , burgemeester2 en1 leden1 van1 de1 bestendige1 deputatie2 dof afsteken4 . John Simenon neemt namens zijn vader de honneurs in ontvangst en zal
    allow, and because of which the suits of governor, mayor and members of the permanent deputation constrast dull(y). John Simenon takes the honors in name of his father and will

The other three tokens were annotated by three annotators: (25) and (26) also showed doubt between dof_1 ‘dull-visual’ and dof_3 ‘energy’, with geen ‘none of the above’ or dof_4 ‘dull-mental’ as third alternative respectively, and (27) between dof_3 ‘energy’, dof_4 ‘dull-mental’ and geen ‘none of the above’.

Example (25) is particularly interesting because the same concordance showed up three times in the same batch (one of which without the text on the left) and only one annotator noticed. This annotator consistently assigned geen ‘none of the above’, first with a long explanation of their reasoning, saying that dof meant “without the shine of a famous life” ant that is not covered by the offered senses, then only mentioning that the same concordance had already come up.4 The other two annotators did not comment anything: one always chose dof_3 ‘energy’ and the other one, twice dof_3 ‘energy’ and this once dof_1 ‘dull-visual’.

  1. leven leven . Rochefort verlangt1 naar groots2 en meeslepend2 , Halliday naar kalm2 en dof . Leconte weet het zo te filmen dat een bank beroven saai1 lijkt en
    live […] life. Rochefort desires [something] big and compelling, Halliday [something] calm and dull. Leconte knows how to film it so that to rob a bank seems boring and
  2. ’ Gracieus en koninklijk , speels en rebels , en daaroverheen soms de doffe glans3 van1 een bijna erotische tristesse2 . ’ Over wie gaat dit citaat van
    ’ Graceful and royal, playful and rebellious, and over that sometimes the dull shine of an almost erotic sadness.’ Who is this quote from […] about
  3. hun gelukkige dagen mogelijk terugkeren . De blauwe ogen van gisteren illustreren dat de doffe Poetin-consensus2 na anderhalf jaar op zijn eind loopt . Nu de regering , na
    possible to come back to their happy days. The blue eyes from yesterday illustrate that the dull Putin-consensus comes to an end after a year and a half. Now the government, after

Adverbial uses

There are four tokens where the majority of the annotators agreed on assigning the dof_3 ‘energy’ tag but some other annotator (not always the same) preferred dof_2 ‘dull-sound’, and where the target is actually an adverb applied to a speech verb: vragen ‘ask’ in (28), zeggen ‘say’ in (29) and klinken ‘sound, say’ in the other two (these from the same batch). There were also other cases with this sort of confusion but they were not adverbial uses: (32) and (33) are described elsewhere and the other two are not worth discussing (nothing to do with sounds).

  1. wat ik1 van de zaak Bevrijdingspop-Balkenende vind . Help1 me1 even , vraag3 ik2 dof . Enfin , Balkenende mag van D66 en de SP natuurlijk Bevrijdingspop in Haarlem
    what I think of the Independence Festival-Balkenende case. Help me for a moment, I ask dull(y). In any case, obviously Balkenende is allowed by D66 and SP […] the Independence Festival in Haarlem
  2. motte ‘t zellef weten hoe ze met me wille , de here’ , zegt1 ze dof als ze na1 het doodsbericht2 van haar zonen ook nog eens krijgt te horen dat ze
    he has to know what he wants of me, Lord’, she says dull(y) as if she still could hear after her sons’ obituaries that she
  3. " De term offday dekt de lading niet , " klonk3 een gekwelde1 Johan1 Bailliu dof . " We probeerden gesloten te voetballen , want wie Gistel ruimte gunt ,
    " The term offday does not cover it, " said a tormented Johan Bailliu dull(y). "We tried to play closed, because whoever gives space to Gistel,
  4. hij uitgelaten1 en vrolijk1 , alsof hij het vooraf wist . Nu klinkt3 hij1 dof . Want hij weet niet wat komt . door Frank BUYSE Al
    elated and cheerful, as if he new beforehand. Now he sounds dull. Because he doesn’t know what is coming. by Frank BUYSE

There were also three cases where the majority sense was dof_2 ‘dull-sound’, the alternative dof_3 ‘energy’, and the actual use adverbial. There is no guarantee that these are the only adverbial uses, but it is quite clear that they were never identified as such.

Interesting cases

There is a number of cases where the majority of the annotators assigned the dof_3 ‘energy’ tag but another annotator suggested an alternative that, upon revision, points to a rather interesting confusion.

In examples (32) and (33) the alternative was dof_2 ‘dull-sound’. They come from the same batch and the dissenter is in both cases the same annotator, but absolutely different confidence ratings in each case: 1 for the disagreeing annotation against 5 in (32) and 4 by all annotators in (33).

  1. Zonder die borden kunnen we niet verder leven . Man1 :1 (1 dof )1 We hebben al borden . Vrouw : Maar niet deze .
    Without those plates we cannot live anymore. Man:(dull) We already have plates. Woman: But not these.
  2. uur . Routinieuze presentatie . Eindeloze stoeten van bloedserieuze2 auteurs die op doffe toon3 voorlezen1 uit eigen werk . Dichtgeknepenbillenmuziek in de geest van Bob Dylan .
    hour. Routine presentation. Endless processions of dead serious authors reading aloud their own work in adull tone. Music of pinched ass?? in the spirit of Bob Dylan.

In examples (34) through (37) the alternative is the literal sense, dof_1 ‘dull-visual’, which is a rather unexpected confusion. Almost all annotations have confidence of 4 or 5, except for one of the agreeing annotations in (34) through (36). (36) and (37) were annotated by the same annotators but the dissident was different in each case (and there was yet another case of "de ogen staan dof’ ‘the eyes are/look dull’, from the same batch and with parallel annotation to (37)), and the other two were assigned to yet other annotators.

  1. . ’ Bij het idee - hijzelf als onaanzienlijk vrouwspersoon - werden zijn ogen2 dof van zelfmedelijden3 . Maar ze lichtten1 weer op bij de volgende zin : ’
    .’ From the idea -himself as an insignificant woman- his eyes went dull out of self-pity. But they lighted up again with the following sentence:
  2. je dat het geheel modern wordt ; in2 die2 tijd2 was het allemaal heel zwaar2 en2 dof . Het wordt nu ook veel spontaner1 opgebracht , al vind ik dat een
    that the whole becomes modern; in those times it was all very heavy and dull. A lot is produced more spontaneously too, although I find it a
  3. van de Real Republicans1 uit Freetown , is van een talentvolle zwarte parel1 verworden1 tot1 een doffe tweedehands , die tegen elk aannemelijk bod1 bereid is te tekenen bij een club1 .
    from the Real Republicans from Freetown, [it] has turned from a talented black pearl into adull secondhand, read to sign into a club at any plausible offer.
  4. kan de plotselinge dood van haar zoon maar niet verwerken . Haar1 ogen3 staan2 dof . Lachen gaat Joanne nog slecht af . Het is meer een
    but cannot process the sudden death of her son. Her eyes are/look dull. Joan still cannot really laugh. It is rather a

A final interesting case is (38), where most of the annotators assigned geen ‘none of the above’ and suggested “flavourless” as alternative meaning, and the third annotator assigned dof_3 ‘dull-energy’, although commenting that there was not enough context. It is the only case where the majority sense is not_listed. It seems like a nominal use??


  1. ‘iron’ wants to try the combination with beef (think: meatball) and that dull, slightly sweet leads by itself to a good potato, boiled or baked.

Nephology of dof

Clouds

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of dof created on 25/03/2020, modeling between 314 and 319 tokens.The stress value of the MDS solution for the cloud of models is 0.132.

The configuration of the cloud of clouds is very similar to the one for heilzaam: two main areas along the vertical axis separated by FOC-WIN, and in each section, from left to right: a group for PPMI:weight, split orthogonally by FOC-POS and SOC-WIN, one for FOC-POS:nav and one for FOC-POS:all, these split orthogonally by PPMI and SOC-WIN (Figure 19).

Figure 19. Clouds of models of 'dof'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=dof'> here</a>.

Figure 19. Clouds of models of ‘dof’. Explore it here.

PPMI really makes an impact in the difference made by other parameters, as can be seen in Figure 20. Therefore, to compare the effect of the stronger parameters, the following selection will be used: PPMI:weight | PPMI:no + LENGTH:FOC + SOC-WIN:4 + SOC-POS:nav.

Figure 20. Distances between models of 'dof' varying along only one parameter, colored by `PPMI`.

Figure 20. Distances between models of ‘dof’ varying along only one parameter, colored by PPMI.

Few tokens are lost by the restrictions on first order context words: 1 with FOC-POS, 2 if combined with one more restriction, and 5 when all three are applied. The distance matrix between the selected models shows that the loosest one has the largest differences with the rest (mostly above 0.5), while FOC-POS barely makes a difference of 0.04 with PPMI:weight (as can be deduced from Figure 20). The situation is very similar when PPMI:no is replaced by PPMI:selection.

The MDS solution shows models very similar to each other, and a very curious thing is that three tokens are outliers in certain models, but not in the stricter ones: they are discussed in the “outliers” subsection. The PPMI:weight models already seem to have two or three distinct groups of tokens before color coding, which are the confirmed to belong to each of the main senses. Even in the PPMI:no models, the distinction is rather neat. One could expect the dof_4 ‘dull-mental’ tokens to mix with the dof_3 ‘dull-energy’ ones, but they do not seem to have a preference for any given group, or not across models in any case.

The t-SNE solutions show two to four main groups in any perplexity (when perplexity is 5, only in PPMI:weight models), even before color coding. Across perplexities the groups are more neatly separated in PPMI:weight and quite decent with FOC-POS:nav | FOC-WIN:10 + PPMI:no. A small denser cluster can be seen across all models, especially with perplexity of 30, next two two or three bigger ones in PPMI:weight | FOC-POS:nav models. Perplexity 50 doesn’t add much: the PPMI:weight models keep some separability and the rest are mere masses.

Color coding lets us see that tokens of the same sense do tend to stick together, except for the less frequent dof_4 ‘dull-mental’. The tighter cluster is made of dof_3 ‘dull-energy’ tokens and the two bigger clouds are dof_1 ‘dull-visual’ and dof_2 ‘dull-sound’ respectively. There are also some rebellious dof_3 ‘dull-energy’ tokens spread around with the other senses or bridging the gap between its native island and the mainland, so to speak. The perplexity 30 solutions show some subgroups within the dof_1 ‘dull-visual’ and dof_2 ‘dull-sound’ as well. Models with PPMI:selection look better than their PPMI:no counterparts, although their separability is no match to PPMI:weight (Figure 21). Other parameters make neglectable differences.

Figure 21. Tokens of 'dof' in the t-SNE solutions (perplexity 30) of the selected models

Figure 21. Tokens of ‘dof’ in the t-SNE solutions (perplexity 30) of the selected models

To further inspect “dof” it could be interesting to look at FOC-WIN:10 + LENGTH:FOC + SOC-POS:nav + SOC-WIN:4 with FOC-POS:nav and/or PPMI:weight.

Outliers

Three tokens of dof exhibit an outlier behaviour in the MDS solutions. They are reproduced here with the context words selected by those models in italics; values in superscript, if included, are PPMI values. (39), already seen in (29), is a rather strong outlier in all models except for those with FOC-POS:all + PPMI:no | PPMI:selection, where it sits comfortably in the middle of the clouds. The relatively rare noun here ‘Lord’ seems to heavily determine the vector of this token. The words in italics in this example are the nouns, adjectives and verbs.

  1. motte ‘t0.083 zellef weten hoe0.28 ze0.166 met0.326 me0.266 wille , de here4.41’ , zegt ze0.166 dof als ze0.166 na0.2 het0.083 doodsbericht van haar0.684 zonen0.299 ook nog0.03 eens0.18 krijgt te horen2.218 dat ze0.166
    he has to know what he wants of me, Lord’, she says dull(y) as if she still could hear after her sons’ obituaries that she

Example (40) is only an outlier (but stronger than the previous one) in FOC-POS:nav + FOC-WIN:10 + PPMI:no, since any combination of FOC-POS:nav and another restriction removes all context words from the token and the FOC-POS:all models don’t see it as an outlier at all. The words in italics are nouns, adjectives and verbs.

  1. het nippertje misging . Een maaibeweging werd haar rechterknie fataal . Een doffe knak , een pijnscheut , Vandecaveye wist meteen dat het niet goed zat .
    [at] the last moment went wrong. A mowing movement was fatal for her right knee. A dull crack, a shooting pain, Vandecaveye knew immediately that it did not look good.

Finally, (41) is an outlier in models with PPMI:selection + FOC-POS:all | FOC-WIN:10 (and to a lesser degree with FOC-POS:all + FOC-WIN:10 + PPMI:weight). All possible context words are highlighted for this example.

  1. je0.326 je0.326 nog0.03 van de ontploffing ? " Bitter weinig0.596 . Twee doffe slagen2.061 , meer niet . Ik0.18 was0.052 die0.051 zondag bezig met0.326 het0.083 plaatsen van
    still from the explosion? Very little. Two dull beats, no more. I was busy that Sunday with moving

hachelijk

The adjective hachelijk was tagged with 2 definitions, reproduced in Table 13. The meaning is basically “hazardous, dangerous”, and can either refer to the danger of something going wrong (hachelijk_1) or, metonymically, to the critical state of something that already did (hachelijk_2). The former was slightly more frequent than the latter in the pilot study, and almost no adverbial or confusing cases were registered.

Table 13. Definitions of ‘hachelijk’.
code definition example freq
hachelijk_1 met kans op een ongunstige afloop, (potentieel) gevaarlijk een hachelijke onderneming 23
hachelijk_2 (re�el) gevaarlijk, netelig, kritiek, benard een hachelijke situatie 16

Sense distribution

The sample consists of 240 tokens (6 batches) out of 1307 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 22. The distributions of the annotations (not majority senses) by annotator are shown in Figure 23. No batch was annotated by 4 annotators. The overall distribution and on each batch is relatively similar to the expectation, although with more cases of hachelijk_2 ‘critical’ than of hachelijk_1 ‘dangerous’. There are only two cases with no agreement, which is to be expected when there are more annotators than senses and their tendency to assign geen ‘none of the above’ tags to adjectives seems rather low.

Figure 22. Distribution of majority senses of 'hachelijk' per batch

Figure 22. Distribution of majority senses of ‘hachelijk’ per batch

Figure 23. Distribution of sense annotations of 'hachelijk' per annotator, grouped by batch.

Figure 23. Distribution of sense annotations of ‘hachelijk’ per annotator, grouped by batch.

“hachelijk” is an adjective with two senses of similar frequency and a subtle meaning difference based on metonymy.

Confusion matrix

The confusion matrix between the majority senses and other tagged senses can be seen in Table 14 (raw number of tokens with such senses assigned) and Table 15 (mean confidence of such sense annotation in each token). The meaning distinction is rather subtle: it is a matter of perspective that may be more or less clear in the context, and the difference between the category of nouns that each sense would prefer is not so clear cut. Therefore, a rather high confusion is expected, and is confirmed by the high numbers in the confusion matrix and by the low percentage of tokens with full agreement (0.54 for hachelijk_1 and 0.62 for hachelijk_2).

Table 14. Non weighted sense matrix of ‘hachelijk’ senses. Proportion of tokens with full agreement per sense-tag is: hachelijk_1: 0.54, hachelijk_2: 0.62.
senses hachelijk_1 hachelijk_2 not_listed unclear wrong_lemma
hachelijk_1 105 47 0 1 0
hachelijk_2 47 133 0 2 1
no_agreement 2 2 1 0 1
total 154 182 1 3 2

The mean confidence values remain above 3 in general, but barely scratch 4, except for the wrong_lemma annotation.

Table 15. Weighted sense matrix of ‘hachelijk’ senses. Mean confidence across the lemma is 3.92; values above are darker and boldened. Median confidence across the lemma is 4.
senses hachelijk_1 hachelijk_2 not_listed unclear wrong_lemma
hachelijk_1 3.81 3.62 0 4 0
hachelijk_2 3.68 4.03 0 1 3
no_agreement 4 3 4 0 5

The cases with no agreement are reported in (42) and (43). In all three the disagreement was between hachelijk_1 ‘dangerous’, hachelijk_2 ‘critical’ and geen ‘none of the above’ (there were no other options, after all). They were part of the same batch and the same annotator chose hachelijk_1 ‘dangerous’ for both. For (42), the annotator that chose geen ‘none of the above’ suggested the meaning of bedenkelijk ‘questionable’??. For (43), instead, that annotator remarked that the target was not being used as an adjective and that it had a strage position. Because it is the only lexical item in its sentence, the models currently analized discarded it.

  1. van 2000 werd gepubliceerd , dan is het duidelijk dat het gebruikmaken van recente groeicijfers een hachelijke zaak is . Waar het om gaat , zijn de maandelijkse werkloosheidscijfers , de
    of 2000 was published, then it’s clear that using the recent growth numbers is a dangerous issue/business. What it is about is the monthly unemployment numbers, the
  2. met een groot aantal helikopters voor charteropdrachten vliegt , zal de toestellen ook onderhouden . Hachelijk Een vlieger , die anoniem wil blijven , stelt dat het opzetten van lijndiensten economisch
    flies with a great number of helicopters with charter assignments, will also maintain the vessels. Hazardous An aviator that wants to remain anonymous states that the goals of shipping services […] economically

Nephology of hachelijk

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of hachelijk created on 25/03/2020, modeling between 238 and 239 tokens.The stress value of the MDS solution for the cloud of models is 0.126.

The subclouds in the cloud of models shows a strong interaction between PPMI, FOC-WIN and FOC:POS. There are five clear subgroups in two main stripes running from the bottom left to the upper right. The leftmost group is made of PPMI:weight models, cut orthogonally by the other two parameters: FOC-WIN horizontally, with FOC-WIN:5 to the left, and FOC-POS vertically, with FOC-POS:nav on top. For the other four groups, the top subclouds are FOC-POS:nav and the left subclouds are FOC-WIN:5, each split by PPMI (Figure 24). There are also some minor LENGTH:FOC groupings.

Figure 24. Clouds of models of 'hachelijk'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=hachelijk'> here</a>.

Figure 24. Clouds of models of ‘hachelijk’. Explore it here.

Like with previous adjectives, PPMI really makes an impact in the difference made by other parameters, as can be seen in Figure 25. Therefore, to compare the effect of the stronger parameters, the following selection will be used: PPMI:weight | PPMI:no + LENGTH:FOC + SOC-WIN:4 + SOC-POS:nav.

Figure 25. Distances between models of 'hachelijk' varying along only one parameter, colored by `PPMI`.

Figure 25. Distances between models of ‘hachelijk’ varying along only one parameter, colored by PPMI.

Only one token is lost, by all FOC-POS:nav models. The distance matrix shows the greatest values for the model with loosest parameters and very low values (0.04 and 0.06) between weighted models that only differ in FOC-POS.

In the MDS solutions, the loosest model has a dense core with a couple of outliers: the structure is a bit more lax with other PPMI:no models and very different with PPMI:weight, where there seems to be five or six dense and separated groups towards the center with some tokens spread around, this being more clear with stricter restrictions. With color coding the senses seem to cluster together, although it’s more clear in the more compact models. The picture looks a bit more clear when only the tokens with full agreement are selected.

In the t-SNE solutions, PPMI:weight models with perplexity 5 have about 5 or 6 dense clusters and the loosest model has one bigger mass with an outlier, the other PPMI:no models showing something in between. Higher perplexities do give a bit more structure: five tight clusters with some wanderers in between for the strictest model, one tight small cluster and a wider mass for the loosest, and something in between for the rest. At least one cluster stays quite firm across perplexities, but the wanderers spread farther and wrap the other clusters as perplexity increases.

Color coding (Figure 26) shows the bigger difference between PPMI:weight and PPMI:no models. This more resilient cluster in PPMI:weight models is one with hachelijk_1 ‘dangerous’ tokens, while the tighter cluster in PPMI:no models is one of hachelijk_2 ‘critical’. PPMI:weight models tend to have three or four very tight, small clusters of hachelijk_2 ‘critical’ interspersed with wanderers of both senses, and this clearly separate cluster of hachelijk_1 ‘dangerous’ with one hachelijk_2 ‘critical’ token: these are cases of hachelijke onderneming ‘dangerous enterprise’, which is actually the example for hachelijk_1 ‘dangerous’ so it’s strange that one occurrence was tagged mainly as hachelijk_2 ‘critical’. PPMI:no models paint a different picture: the tighter cluster is made of cases of “iemand uit wijn hachelijke positie bevrijden” ‘free someone from a critical position’, which also cluster nicely in the PPMI:weight models. The other two clusters of hachelijk_2 ‘critical’, one for “hachelijk avontuur” ‘dangerous adventure’ and one for “hachelijke situatie” ‘critical situation’, can also be seen in PPMI:no + FOC-POS:nav + FOC-WIN:5 and a bit less clearly in less strict models, but not at all in the loosest model. The tokens of “hachelijke onderneming” ‘dangerous enterprise’ do stick together in the PPMI:no models but don’ stand out from the rest of the crowd. PPMI:selection models don’t show that cluster as such either, but at least seem to keep the two senses further apart than their PPMI:no counterparts.

Figure 26. Tokens of 'hachelijk' in the t-SNE solutions (perplexity 30) of the selected models.

Figure 26. Tokens of ‘hachelijk’ in the t-SNE solutions (perplexity 30) of the selected models.

The second order parameters make very little difference, but there is a certain beauty in the model with FOC-WIN:5 + FOC-POS:nav + PPMI:selection + LENGTH:5000 + SOC-POS:nav + SOC-WIN:4 that is not found in other models (Figure 27)

Figure 27. Beautiful token cloud of 'hachelijk'.

Figure 27. Beautiful token cloud of ‘hachelijk’.

For further inspection it would be interesting to look into Figure 27 and maybe models that differ in one parameter from it.

geestig

The adjective geestig was tagged with 2 definitions, reproduced in Table 16. The meaning is basically “witty and funny”, and the different senses pertain to it’s application to people (geestig_1) or to the products and expressions of people, such as books, glances, remarks (geestig_2). In the pilot study the former was much less frequent than the latter and there were very few confusing or adverbial cases.

Table 16. Definitions of ‘geestig’.
code definition example freq
geestig_1 scherpzinnig en humoristisch van aard een geestige collega 6
geestig_2 blijk gevend van, uitdrukking gevend aan, gekenmerkt door scherpzinnigheid en humor een geestig boek, een geestige blik, een geestige opmerking 32

Sense distribution

The sample consists of 280 tokens (7 batches) out of 3970 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 28. The distributions of the annotations (not majority senses) by annotator are shown in Figure 29. Batch 7 was annotated by 4 annotators. The distribution across batches and overall are very similar to the estimation, maybe a bit less skewed. No tokens had a geen ‘none of the above’ tag as the majority sense.

Figure 28. Distribution of majority senses of 'geestig' per batch

Figure 28. Distribution of majority senses of ‘geestig’ per batch

Figure 29. Distribution of sense annotations of 'geestig' per annotator, grouped by batch.

Figure 29. Distribution of sense annotations of ‘geestig’ per annotator, grouped by batch.

“geestig” is an adjective with two senses of very uneven distribution, the metonymic extension being much more frequent than the core, anthropocentric sense.

Confusion matrix

Matrices

The confusion matrix between the majority senses and other tagged senses can be seen in Table 17 (raw number of tokens with such senses assigned) and Table 18 (mean confidence of such sense annotation in each token). Given the metonymy that links the senses of this adjective, we could expect some confusion between the senses, especially in cases with less clear contexts, but at the same time the clear cut category of “people” for the objects of geestig_1 ‘witty-person’ could be grounds to less confusion for tokens with this sense as the majority.

The number of tokens with confusion is indeed quite high, but the percentage of tokens with full agreement is actually lower for geestig_1 ‘witty-person’ (0.55) than for geestig_2 ‘witty-expression’ (0.62).

Table 17. Non weighted sense matrix of ‘geestig’ senses. Proportion of tokens with full agreement per sense-tag is: geestig_1: 0.55, geestig_2: 0.62.
senses geestig_1 geestig_2 not_listed unclear wrong_lemma
geestig_1 83 34 0 1 2
geestig_2 69 191 2 1 1
no_agreement 6 6 1 2 0
total 158 231 3 4 3

The confidence for the annotations is rather low (although normally above 3, so in the upper half), except for the two tokens with geestig_1 ‘witty-person’ as majority sense and wrong_lemma or unclear as alternative. These tokens are discussed in “Wrong lemma” along with others that received a geen ‘none of the above’ tag, while the six with no agreement in “No agreement”.

Table 18. Weighted sense matrix of ‘geestig’ senses. Mean confidence across the lemma is 3.56; values above are darker and boldened. Median confidence across the lemma is 4.
senses geestig_1 geestig_2 not_listed unclear wrong_lemma
geestig_1 3.6 3.35 0 4 4.5
geestig_2 3.17 3.62 3 2 1
no_agreement 2.17 3.25 3 0 0

Wrong lemma

There are 10 tokens that received some geen ‘none of the above’ tag (none that had it as majority sense), 3 of which showed no agreement between the annotators and will therefore be described in this subsection. Context words selected as helpful by the annotators will be boldened and the number of annotators that selected them reported in superscript.

Examples (44) and (45) belong to the same batch: they were annotated by the same annotators. Two of the annotators assigned geestig_1 ‘witty-person’ to these concordances, one with confidences of 5 (maximum) and 3 and the other one with minimum confidence each time. The other annotator assigned geen ‘none of the above’ with confidence of 4 to both, reporting that there was not enough context to deduce the right part of speech in (44) and that the target was actually an adverb in (45), which is correct.

  1. Giddins en Mandel2 geblinddoekt en met één hand op de rug gebonden een stuk beter en geestiger dan1 de1 romancier1 , die hier grossiert in houterige zinnen , gemeenplaatsen en gezochte , het
    Giddis en Mandel blindfolded and with one hand tied at the back a bit better and funnier than the novelist, who spreads here wooden sentences, common places and quests??, the
  2. verkeerd spoor zetten . Over het algemeen schrijft2 Loudon2 heel toegankelijk1 , soms heel geestig en altijd als iemand die het leuk vindt om zo sjeuïg mogelijk een verhaal te vertellen
    the wrong path. In general Loudon writes very accessibly, sometimes very witty and always as someone who likes to tell a story as beautifully as possible

Examples (46) and (47) belong to another batch but the disagreeing annotator was different each time. For (46) the majority sense was geestig_2 ‘witty-expression’, with medium confidences, and the other annotator reported a wrong part of speech; with low confidence but correctly. For (47) the majority sense was geestig_1 ‘witty-person’, also with medium confidences, and the other annotator stated with maximum confidence, and correctly, that it was actually a noun.

  1. weg en vindt zichzelf de meest fantastische kerel . Dan Sonja … Geestig hoe1 zij1 zich heeft ontwikkeld1 . Jarenlang een talkshow voor een miljoenenpubliek en nu
    and considers himself the most fantastic guy. Then Sonja… Funny how she has developed. For years a talkshow for an audience of millions and now
  2. grens op tussen het2 hilarische en het onsmakelijke , het komische1 en het walgelijke1 , het geestige en het psychopatische2 . En gaat er overheen . Net als zijn1
    border between the hilarious and the distasteful, the comical and the disgusting, the witty and the psychopatic. And goes over it. Just like his5

Examples (48) and (49) come from another batch and this time again the dissident annotator is different each time. In both of them the majority sense is geestig_2 ‘witty-expression’, with high confidence except for one of the annotators in the latter. For (48), the other annotator suggested with high confidence that the usage in this concordance might come from dialect, meaning “leuk, schattig” ‘cute’. For (49), instead, the disagreeing annotator pointed to a general lack of context.

  1. in de buurt bent , is een bezoekje leuk . Vooral de patisserieafdeling3 is geestig . De rest is een beetje kneuterig . Hattem is een schattig
    are in the neighborhood, a visit is nice. Mainly the pastry shop is nice. The rest is a bit cozy??. Hattem is a cute
  2. Van de Vendel slaagde er bijzonder goed in om de ontwapenende kinderlijke logica1 geloofwaardig , geestig én met1 veel1 gevoel1 over2 te1 brengen2 . Bovendien zitten de versjes erg goed
    Ven de Vendel managed particularly well to communicate the disarming childish logic believably, with humour/wit and with a lot of feeling. Besides the verses look very well

Example (50) belongs to the batch annotated by four annotators, where half of the tokens with no agreement come from. Here, two of the annotators agreed on geestig_2 ‘witty-expression’ (one with maximum confidence and the other one with only 1), another one suggested geestig_1 ‘witty-person’ and another, the sense of “aangenaam” ‘pleasant’, which could be linked to (48).

  1. vindt het2 werk1 alleszins geslaagd1 . " Ik1 vind1 het1 heel leuk2 , heel1 geestig " , reageert Marij Wijnants . Gevraagd naar de betekenis van het werk antwoordt
    considers the work in any case successful. “I think it is very nice, very witty”, comments Marij Wijnants. Asked about the meaning of the work [she] answers

No agreement

In 6 cases there was no majority sense: in the first three reported here each annotator chose a different sense or the geen ‘none of the above’ tag, while in the second half there were four annotators and each half of them chose one or the other sense. In these examples the context words selected by the annotators are boldened and the number of annotators that chose them will be reported in superscript.

In examples (51) and (52) the noun modified by the target is rather unexpected: arrogantie ‘arrogance’ and stedenbouwproject ‘urban development project’ respectively. In the former, the annotator that chose the geen ‘none of the above’ tag reported that the sense was very ambiguous; in the latter (a different annotator), that it did not mean “humoristisch” ‘humorous’ in this context. Unlike their colleagues, the annotators that selected the geen ‘none of the above’ tag in each case selected no context words as useful (which is kind of expected).

  1. Stel je voor dat ze verloor . Een nog vroeger voorbeeld van haar geestige arrogantie2 : ze dwong haar eerste en enige echtgenoot , met wie ze van 1928 tot
    Imagine if she lost. An even earlier example of her witty arrogance: she forced her first and only spouse, with whom she […] from 1928 to
  2. is er een stedenbouwproject2 van1 Bodys1 Isek1 Kingelez1 voor1 Kinshasa1 in1 het1 jaar1 30001 , zo geestig en kleurrijk2 dat je zou willen dat alle bloedserieuze architecten en designers hier verplicht naar zouden
    there is an urban development project by Bodys Isek Kingelez for Kinshasa in the year 3000, zo witty and colorful that you would want that all the dead serious architects and designers here were compelled

Example (53) was assigned geestig_1 ‘witty-person’ by one annotator with minimum confidence and an extra comment reporting confusion, geestig_2 ‘witty-expression’ by another one, with high confidence, and geen ‘none of the above’ by the third one, reporting insufficient context. Since the target is the only word in its sentence, it has no context words in the current model and cues won’t be registered for now.

  1. haar ’ echte’dagelijkse leven tijdens de avonden in Stockholm . Geestig . De botsing van nep en echt . Indruk in Arnhem maakt wél Amie Dicke
    her ‘real’ daily life during the evenings in Stockholm. Witty. The clash of fake and real. Amie Dicke does make an impression in Arnhem

Examples (54) and (55) also have atypical nouns modified by the target: spelletje ‘little game’ and wat ik breng ‘what I bring’ respectively. In both cases, the same two annotators chose geestig_1 ‘witty-person’ with low-medium confidence (one was high in the second concordance) and the other two chose geestig_2 ‘witty-expression’, one with maximum confidence and the other with a low one.

For (54), all of them tagged spelletje ‘little game’ as an important context word; the ones that chose geestig_1 ‘witty-person’ also selected grapjas ‘joker’ and one of them solist ‘soloist’, dirigent ‘conductor’ and pianist ‘pianist’ as well. For (55), the ones that chose geestig_2 ‘witty-expression’ chose wat ik breng ‘what I bring’ as relevant context words and one of the others chose gewoon (L0), breng (L1), niet (R7), charisma (R11), while the other chose none.

  1. de solist1 de grapjas2 uit . Het1 spelletje4 tussen dirigent1 en pianist1 is2 soms geestig , maar ook ergerlijk . Veronique Rubens Kristien Vermoesen Veronique Rubens
    the soloist […] the joker. The little game between conductor and pianist is sometimes witty, but also annoying. Veronique Rubens Kristien Vermoesen Veronique Rubens
  2. te zijn . Ik ga er namelijk van uit dat1 wat2 ik2 breng3 gewoon2 geestig is1 . Ik moet het niet1 hebben van mijn charisma1 , maar louter van
    be. I start thus by saying that wat I bring is just witty. I do not like my charisma, but louder than

Example (56) was also tagged twice with each sense, but the annotators did not group like in the previous two cases. The geestig_1 ‘witty-person’ annotations had confidences of 5 and 2, and those of geestig_2 ‘witty-expression’, the ones that selected other words beyond het is ‘it is’, including kick ‘kick’, of 5 and 0.

  1. neus van duizend joelende mensen als1 eerste1 boven1 komen1 op de Oude Kwaremont : het4 is4 geestig , geeft1 me1 een1 kick2 . Maar die kick moet veel groter zijn als
    nose of a thousand screaming people as the first come up to the Old Kwaremont: it is funny, [it] gives me a kick. But that kick must be much bigger than

Nephology of geestig

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of geestig created on 25/03/2020, modeling between 266 and 276 tokens.The stress value of the MDS solution for the cloud of models is 0.176.

The main division of the cloud of models is made by FOC-WIN, along the vertical dimension. Each hemisphere has one clear FOC-POS:nav subcloud down to the left and a longer, disperser FOC-POS:all on top towards the right, each group with vertical stripes of PPMI (with PPMI:selection between the other two) and split orthogonally by SOC-WIN (Figure 30)

Figure 30. Clouds of models of 'geestig'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=geestig'> here</a>.

Figure 30. Clouds of models of ‘geestig’. Explore it here.

As illustrated in Figure 28, Figure 29, Figure 30, Figure 31,FOC-POS (less with PPMI:weight), FOC-WIN and PPMI (between PPMI:weight and the others) make the most difference when all other parameters are fixed, so for the first analysis the selection will keep SOC-POS:4 + SOC-WIN:nav + LENGTH:FOC constant, discarding PPMI:selection.

Figure 32. Distances between models of 'geestig' varying along only one parameter, colored by `PPMI`.

Figure 32. Distances between models of ‘geestig’ varying along only one parameter, colored by PPMI.

The different restrictions to first order context words lose 1 to 10 tokens. The distance matrix shows that the loosest model is the most different to all, with distances between 0.34 and 0.69, but also that the distances between the models are rather high, with a minimum of 0.18 and most values above 0.3.

MDS solutions have a main core with satellites. The loosest model has a more compact core with 5 outliers: one is lost by PPMI:weight models, another by FOC-POS:nav models and the others remain at least peripheral in all models. Two of them have very short sentences, which in these sentence bound models imply an already strict restriction of the context words. Color coding suggests a relationship of inclusion: geestig_1 ‘witty-person’ seems to tend to stick together in one half of the plot but geestig_2 ‘witty-expression’ is all over the place, surrounding it.

The t-SNE models don’t show much structure in terms of clusters (they show a structure of little separability?). With perplexity 5 they have an archipelago shape, except for the FOC-WIN:10 + PPMI:no models, that look more like homogenous masses with at most two groups. With perplexity 20, some PPMI:weight or FOC-POS:nav models seem to have small clusters next to a big disperse mass, but in general no separation between subclouds can be detected. This is even more the case with higher perplexity, where the FOC-WIN:10 + PPMI:no models just look more compact (or all the PPMI:no models with perplexity 50) and only the FOC-POS:nav + PPMI:weight models show a suggestion of some clusters.

Whatever clusters can be guessed from a non color coded cloud, coding doesn’t throw any light. If anything, the less frequent geestig_1 ‘witty-person’ seem to stick together in the PPMI:no models but are equally spread in the PPMI:weight ones. PPMI:solution models, on the other hand, do seem to show a bit more separability, at least for most geestig_1 ‘witty-person’ tokens (Figure 33). A more thorough examination of the subclusters and the annotations might clarify whether this represents the actual relationship between the senses or is an artifact of the annotation.

Figure 33. Tokens of 'geestig' in the t-SNE solutions (perplexity 20) of the selected models.

Figure 33. Tokens of ‘geestig’ in the t-SNE solutions (perplexity 20) of the selected models.

The second order parameters make neglectable differences.

For further inspection of “geestig” clouds, among the LENGTH:FOC + SOC-POS:nav + SOC-WIN:4 models it could be most interesting to look into those with PPMI:weigth | PPMI:selection + FOC-POS:nav | FOC-WIN:5.

hoopvol

The adjective hoopvol was tagged with 2 definitions, reproduced in Table 19. It basically translates as “hopeful”, with the senses applying to each of two perspectives: to someone that has hope or something that expresses such feeling (hoopvol_1) or to something that gives hope (hoopvol_2), which is half as frequent in the pilot concordance. Half of the instances identified for hoopvol_1 ‘hopeful-feel’ were in the expression “dat stemt mij hoopvol” ‘that gives me hope, that makes me hopeful’, where the adjective modifies the object through a verb, but is still applied to the noun. Some other cases, like “hij kijkt hoopvol omhoog” ‘he looks up hopeful’, could be either interpreted as the target modifying the verb or the subject; with the annotators tendency to assume the target is an adjective, that is probably not going to be an issue.

Table 19. Definitions of ‘hoopvol’.
code definition example freq
hoopvol_1 (van personen, uitingen, gedragingen etc.) blijk gevend van hoop, vol hoop, optimistisch een hoopvolle stemming, dat stemt mij hoopvol 25
hoopvol_2 reden tot hoop gevend, beloftevol hoopvolle perspectieven 12

Sense distribution

The sample consists of 240 tokens (6 batches) out of 3680 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 34. The distributions of the annotations (not majority senses) by annotator are shown in Figure 35. No batch was annotated by 4 annotators. The distribution of the senses is quite stable across batches, and if what was tagged as geen ‘none of the above’ in the pilot concordance is accepted as a case of hoopvol_1 ‘hopeful-feel’, then the overall distribution is exactly like predicted. There were very few cases with no agreement and one where the majority assigned the geen ‘none of the above’ tag.

Figure 34. Distribution of majority senses of 'hoopvol' per batch

Figure 34. Distribution of majority senses of ‘hoopvol’ per batch

Figure 35. Distribution of sense annotations of 'hoopvol' per annotator, grouped by batch.

Figure 35. Distribution of sense annotations of ‘hoopvol’ per annotator, grouped by batch.

“hoopvol” is an adjective with two senses, an anthropocentric one and a metonymical extension half as frequent.

Confusion matrix

The confusion matrix between the majority senses and other tagged senses can be seen in Table 20 (raw number of tokens with such senses assigned) and Table 21 (mean confidence of such sense annotation in each token). Given that the relationship between the senses is metonymical, some confusion would be expected, but at the same time the emphasis on the (human) subject may lower it.

There is indeed quite some confusion between the senses, but a relatively high proportion of annotations with full agreement for hoopvol_1 ‘hopeful-feel’. The confidence values are also higher in the agreeing hoopvol_1 ‘hopeful-feel’ annotations, but not so low in the rest.

Table 20. Non weighted sense matrix of ‘hoopvol’ senses. Proportion of tokens with full agreement per sense-tag is: hoopvol_1: 0.81, hoopvol_2: 0.56.
senses hoopvol_1 hoopvol_2 between not_listed unclear wrong_lemma
hoopvol_1 166 30 1 0 1 0
hoopvol_2 27 71 2 1 1 0
wrong_lemma 0 0 0 0 1 1
no_agreement 2 2 0 0 2 0
total 195 103 3 1 5 1
Table 21. Weighted sense matrix of ‘hoopvol’ senses. Mean confidence across the lemma is 3.92; values above are darker and boldened. Median confidence across the lemma is 4.
senses hoopvol_1 hoopvol_2 between not_listed unclear wrong_lemma
hoopvol_1 4.05 3.67 0 0 4 0
hoopvol_2 3.59 3.76 1.5 0 0 0
wrong_lemma 0 0 0 0 4 5
no_agreement 4.5 1.5 0 0 0 0

One token, reproduced in (57), was assigned the geen ‘none of the above’ tag by all three annotators and with high confidence: in the comments, one asked if it wasn’t a proper name, another one stated that it was the name of a team, and the third reported little to no context. This token is not modelled by the current clouds.

  1. Tim ( van ongeva ) 7836 Pets Erik ( desjotters ) 7828 Spitaels Alfons ( hoopvol ) 7772 Vanderbeken Willy ( tomka ) 7750 ZOTTEGEM Van Cauwenberge Jens ( bulls )
    Tim ( from ongeva ) 7836 Pets Erik ( desjotters ) 7828 Spitaels Alfons ( hoopvol ) 7772 Vanderbeken Willy ( tomka ) 7750 ZOTTEGEM Van Cauwenberge Jens ( bulls )

Two tokens received no majority sense: one annotator assigned hoopvol_1 ‘hopeful-feel’‘with high confidence, another one hoopvol_2 ’hopeful-give’ with low or medium confidence and another one geen ‘none of the above’ with minimum confidence. They belong to the same batch and the same annotator assigned geen ‘none of the above’, but the other annotators switched for their annotations, so to speak. The explanation for the geen ‘none of the above’ tag was “rare context” in the case of (58) and “foute Nederlands” ‘wrong Dutch’ in the case of (59).

  1. van het aanbod van SDMI zullen vrijdag ook te lezen staan op www.hacksdmi.org . Hoopvolle hackers zullen tot 7 oktober moeten wachten voor ze kunnen demonstreren hoe ze erin geslaagd zijn
    of the offer of SDMI will be also available on Friday on www.hacksdmi.org. Hopeful hackers will have to wait until October 7 to demonstrate how they managed to
  2. laat dit ons toe de gewestwegenproblematiek gedeeltelijk op te lossen " , heet het bij Pira hoopvol . Verkeersveiligheid is prioritair voor het huidige Mortselse stadsbestuur . Het is
    allows us to partially solve the problematic of the provincial roads", is called around Pira hopeful. Road safety is a priority for the current city counsel fo Mortsel. It is6

Nephology of hoopvol

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of hoopvol created on 25/03/2020, modeling between 228 and 237 tokens. The stress value of the MDS solution for the cloud of models is 0.145.

The main split of the cloud of models is made by FOC-WIN along the vertical dimension. Each hemisphere has two distinct groups, one to the left made of PPMI:weight tokens, orthogonally split by FOC-POS and SOC-WIN, and one for each FOC-POS (with FOC-POS:nav between PPMI:weight and FOC-POS:all), split orthogonally by PPMI (excepto PPMI:weight) and SOC-WIN (Figure 36).

Figure 36. Clouds of models of 'hoopvol'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=hoopvol'> here</a>.

Figure 36. Clouds of models of ‘hoopvol’. Explore it here.

Figure 37 shows that PPMI:weight impacts the difference made by other parameters in models where all other parameters are equal. Because the second order parameters don’t seem to make much of a difference, the first selection of models will include those with LENGTH:FOC + SOC-POS:nav + SOC-WIN:4, excluding PPMI:selection.

Figure 38. Distances between models of 'hoopvol' varying along only one parameter, colored by `PPMI`.

Figure 38. Distances between models of ‘hoopvol’ varying along only one parameter, colored by PPMI.

Two tokens are lost in FOC-POS:nav models: 3 if combined with FOC-WIN:5, 4 in combination with PPMI:weight and 9 with all three restrictions. The distance matrix show that the loosest model is the most different to the rest, with values between 0.40 and 0.79, but also that all PPMI:no models present higher distances than PPMI:weight ones, which show distances between 0.10 and 0.28 to each other (Distance matrix 3).

Distance matrix 3. Distance matrix between some models of ‘hoopvol’
id FOC-POS FOC-WIN PPMI
1 nav 10_10 weight
2 nav 10_10 no
3 all 10_10 weight
4 all 10_10 no
5 nav 5_5 weight
6 nav 5_5 no
7 all 5_5 weight
8 all 5_5 no

The MDS solutions don’t look drastically different from each other. The PPMI:no + FOC-POS:all models look more compact than the others, and the PPMI:weight models seem to have a small cluster of “iemand hoopvol stemmen” ‘make someone hopeful, give someone hope’ slightly distinct from the rest of the cloud. The two senses overlap completely.

The t-SNE models are mere archipelagos with perplexity 5, but with higher perplexity they start distinguishing a tight cluster of “iemand hoopvol stemmen” ‘make someone hopeful, give someone hope’ against the rest of the cloud. This cluster is particularly visible in PPMI:weight models, where it stays outside the main cloud and really stands out with perplexities 30 and 50 (and in retrospective it can already be found with perplexity of 5), but is also present in PPMI:no models, albeit inside the main cloud, except for the loosest one. Further inspection reveals small pockets for “hoopvol uitkijken naar” ‘look forward to with hope’, which is actually an adverbial use, and “hoopvol zeggen” ‘say with hope’. Other than that, there is no sense based grouping of the tokens. PPMI:selection does offer some improvement compared to PPMI:no, but not much (Figure 39).

Figure 39. Tokens of 'hoopvol' in the t-SNE solutions (perplexity 30) of the selected models

Figure 39. Tokens of ‘hoopvol’ in the t-SNE solutions (perplexity 30) of the selected models

LENGTH does not seem to make much of a difference in the distance matrices, but some t-SNE solutions seem to push the “iemand hoopvol stemmen” ‘make someone hopeful, give someone hope’ cluster further from the main cloud with LENGTH:5000 against LENGTH:FOC, and the latter models seem also a bit more disperse in their MDS solutions than their LENGTH:5000 counterparts, so it might be interesting to look into that distinction. Other than that, the second order parameters don’t seem to make much of a difference. Once PPMI:weight is set, not even the other first order parameters make a great difference in de distance matrix, but the resulting models do have slight differences that might be worth looking into.

For further inspection we’ll look into PPMI:weight + SOC-POS:nav + SOC-WIN:4 + LENGTH:5000 | LENGTH:FOC models.

hemels

The adjective hemels was tagged with 2 definitions, reproduced in Table 22. It is translated to “heavenly” or “divine” and the senses correspond to the literal application to something belonging to Heaven (hemels_1) or a broader one to things that are thought as wonderful as if they came from Heaven (hemels_2). The latter is expected to be twice as frequent as the former and few ambiguous or adverbial cases are expected.

Table 22. Definitions of ‘hemels’.
code definition example freq
hemels_1 betrekking hebbend op de hemel de hemelse Vader, de hemelse boodschap 12
hemels_2 verrukkelijk, heerlijk, zalig, goddelijk een hemelse verschijning, een hemelse stem 24

Sense distribution

The sample consists of 240 tokens (6 batches) out of 1417 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 40. The distributions of the annotations (not majority senses) by annotator are shown in Figure 41. Batch 5 was annotated by 4 annotators. The distribution seems quite stable across batches and very similar to the estimate; the sixth batch does have the lowest number of hemels_1 ‘divine’ tokens, the highest number of cases with no agreement and most of those with geen ‘none of the above’ as majority sense.

Figure 40. Distribution of majority senses of 'hemels' per batch

Figure 40. Distribution of majority senses of ‘hemels’ per batch

Figure 41. Distribution of sense annotations of 'hemels' per annotator, grouped by batch.

Figure 41. Distribution of sense annotations of ‘hemels’ per annotator, grouped by batch.

“hemels” is an adjective with two senses, a literal one and an extension based on perceived similarity, the former being half as frequent as the latter.

Confusion matrix

Matrix

The confusion matrix between the majority senses and other tagged senses can be seen in Table 23 (raw number of tokens with such senses assigned) and Table 24 (mean confidence of such sense annotation in each token). In principle, the literal sense is so specific (and likely to occur capitalized) that it should hardly allow any confusion. If any, there should be more cases of the broader hemels_2 ‘wonderful’ as alternative to hemels_1 ‘divine’ than the other way around. That is not really the case (even in relative frequencies, but the difference is very small), but the proportion of tokens with full agreement is higher for hemels_2 ‘wonderful’ than for hemels_1 ‘divine’. The confidence values are also relatively low, although mostly above 3, but only in the agreeing hemels_2 ‘wonderful’ annotations does the mean reach the value of 4. Looking at some concordances suggests that the annotators did not think of the senses in the expected way: hemels_1 ‘divine’ is the most literal and strictest sense, but they tried to discard the possibility of hemels_2 ‘wonderful’ (which is harder to grasp) before applying the first tag, rather than the other way around.

There are also 5 cases with full agreement on the not_listed tag, described in “Alternative senses” and 9 without agreement between the annotators, described in “No agreement”.

Table 23. Non weighted sense matrix of ‘hemels’ senses. Proportion of tokens with full agreement per sense-tag is: hemels_1: 0.66, hemels_2: 0.71, not_listed: 1.
senses hemels_1 hemels_2 between not_listed unclear wrong_lemma
hemels_1 90 21 3 5 3 0
hemels_2 33 136 1 1 4 0
not_listed 0 0 0 5 0 0
no_agreement 4 8 1 5 6 2
total 127 165 5 16 13 2
Table 24. Weighted sense matrix of ‘hemels’ senses. Mean confidence across the lemma is 3.83; values above are darker and boldened. Median confidence across the lemma is 4.
senses hemels_1 hemels_2 between not_listed unclear wrong_lemma
hemels_1 3.81 3.43 0.33 2 1 0
hemels_2 3 4 2 3 3.25 0
not_listed 0 0 0 3.8 0 0
no_agreement 2 2.62 2 2 1.67 3.5

Alternative senses

For five concordances, all three annotators selected the geen ‘none of the above’ tag and suggested the same alternative (or almost). In the first case (example (60)), the comments pointed to a definition on the lines of “in the sky”, as it refers to a satellite; one of the annotators reported maximum confidence, the other two only 3. This concordance belonged to a different batch than the other four, where the annotators unanimously suggested the alternative “(heel) erg” ‘serious, tough’ (or in one case “heel grote” ‘very big’). In almost all cases the annotations had high confidence, with one particular annotator always reporting medium confidence.

  1. Die hemelse weervoorspeller is SOHO , een succesvolle Europees-Amerikaanse zonnesatelliet op anderhalf miljoen kilometer van de aarde .
    That heavenly weather forcaster is SOHO, a successful European-American solar satellite a million and a half kilometers from the Earth.

Examples (61) and (62) are very similar: in both cases the noun modified by the target is schrik ‘terror’, which will clearly not be divine or wonderful, but serious, intense.

  1. kleine jongen ben ik uit een brandend huis moeten vluchten . Ik heb een hemelse schrik voor vuur ! " Die uitleg voorkwam niet dat François D. ( 55
    small boys I had to flee from a bruning house? I’m awfully afraid (lit. I have an heavenly terror) of fire!" That explanation did not prevent François D. (55
  2. geven tot de piste , en dan rechtuit rechtaan te spurten . Ik had hemelse schrik van Hammond . Nadien was Roger wel net zo blij als ik met
    give to the track, and then sprint straight on and ahead. I was really scared of Hammond (lit. had heavenly terror of Hammond). Afterwards Roger was just as happy as me with

For (63) and (64) the same annotators offered the same alternative, and while in principle it is not wrong, here the target has an adverbial use: the objects are zwaar ‘hard, tough’ and afzien ‘suffer, endure’ respectively.

  1. de prijzen gedeeld maar als je met tien punten achterstand de winter ingaat wordt het wel hemels zwaar . Als je bij Berlare speelt weet je dat de druk groot is
    split the prizes but if you start the winter ten points behind it gets really (lit. heavenly) hard. If you play around Berlare you know the pressure is big
  2. Robinson niet tevreden over hoe ze in beeld worden gebracht ? Maar jullie hebben hemels afgezien en jullie zien er zo scherp uit ! De verliezers van The Block
    [Was] Robinson not happy/satisfied with how they were represented/pictured? But you went through hell (lit. have suffered heavenly) and you look so sharp! The losers of The Block

No agreement

There where 9 tokens where the annotations did not agree on any given tag. (65) was annotated by four annotators, split between hemels_1 ‘divine’ and hemels_2 ‘wonderful’ and with low or medium confidence (and one of the annotators that selected hemels_2 ‘wonderful’ actually reported strong heistation), but the rest were annotated by three: (66) and (67) were tagged with both senses and a geen ‘none of the above’ tag and the rest with one of the sense tags and two geen ‘none of the above’ tags but with different explanations.

  1. uit het Spaanse bergdorp . Ver weg van de stress , leven van de hemelse dauw tussen de ruïnes van een godverlaten Spaans bergdorp . Dromen van een Vredesdorp.Het
    from the Spanish mountain village. Very far from stress, life of the heavenly dew among the ruines of a godforsaken Spanish mountain village. Dreams of a Peace village. The

The annotations of (66) and (67) had all low or (in one case) medium confidence; one annotator assigned hemels_1 ‘divine’, another one hemels_2 ‘wonderful’ but commented that there was not enough context for the former and that they didn’t understand the use of the target in the latter, and one geen ‘none of the above’ reporting lack of context. They were different batches.

  1. voert de duistere tekeningen van Piranesi ten tonele als droombeelden van de jonge Quinten van de hemelse ruimte : een plek met alleen een interieur en geen buitenkant . Daarom ben
    executes the dark drawings of Piranesi on the scene as dream scenes of young Quinten from the heavenly space: a place with only an interior and no exterior. Because of that [I] am
  2. de rechten een vermogen kunnen opleveren . Het zijn niet alleen beschilderde bussen en hemelse faxen die de formule uitmaken . Ook het triviale feit dat er bij Jerry
    can deliver the rights and properties. It is not only painted mailboxes and heavenly faxes that make up the formula. Also the trivial fact that with Jerry

The annotations of (68) through (70) had medium or low confidence, except for the hemels_2 ‘wonderful’ annotation of (70), which had maximum confidence. In each case, one annotator assigned hemels_2 ‘wonderful’, and two assigned geen ‘none of the above’, one of which suggested an unlisted sense. For (68) the suggestion was “gelukzalig en rustgevend” ‘blissful and soothing’, while the other annotator reported hesitation between the given options; for the other two, one suggested that both senses were at play simultaneosuly, while the other that there was not enough context.

  1. het zelfs lekkerder dan pindakaas ? ’ wil Fourlani nog van hem weten , terwijl zij hemels naar het plafond kijkt . Pitt draait met zijn ogen , terwijl tranen alweer
    even tastier than peanut butter?’ Fourlani keeps asking him, while she heavenly? looks at the ceiling. Pitt rolls his eyes, while again tears
  2. van de medewerkers . Al deed de VIP-tent van Kiwanis gouden zaken omdat de hemelse watervallen wel noopten om beschutting te blijven zoeken in de dichte omgeving van de toog .
    of the colleagues. Kiwanis’ VIP-tent already did some golden business because the heavenly waterfalls did require shelter to keep looking in the close surroundings of the bar??
  3. Hemelse aardappelen Wie zin heeft in een letterlijk hemelse smaak , kan voortaan ruimte-aardappelen kopen
    Heavenly potatoes Whoever feels like tasting a literally heavenly flavour, can buy from now on space-potatoes

Example (71) comes from the same batch as (69) and (70): the same annotator that suggested both meanings at play did the same for this concordance, while another one reported lack of context and the third one assigned hemels_1 ‘divine’ with high confidence.

  1. , naar China om orde op zaken te stellen . Nero 161 : De hemelse vrede . Standaard Uitg. , 39 blz. , 4,60 euro De Schemerzwervers
    , to China to put things in order. Nero 161: The heavenly peace. Standaard Pub:, 39 pp., 4.60 euro The Twilight Wanderers

Finally, in (72) and (73) one annotator assigned hemels_2 ‘wonderful’, one correctly pointed out that it was the wrong part of speech, and one justified their geen ‘none of the above’ tag in another way: reporting insufficient context for (72) and suggesting another sense for (73), namely “heel erg hard” ‘really very hard’.

  1. ! en de DDD-speler staat na damafname met 42-37 zeker niet minder . Groot - Hemels 2-0 ZDC ’t Zand - DDD Alkmaar D3 : 42 …. 9-14 ? Ook 8-12
    ! and the DDD player stands with 42-37 after capture certainly no less. Great - Heavens 2-0 ZDC the Sand - DDD Alkmaar D3: 42… 9-14? Also 8-12
  2. Hij merkte het niet eens , maar hij werd meegesleurd door die zee , waarvan hij hemels kon genieten . Soms maakte hij een afgietsel van hoe hij die zee zag
    He didn’t even notice, but he was dragged along by that sea, which he could enjoy a lot (lit. heavenly). Sometimes he made a cast of how he saw the sea

Nephology of hemels

Clouds

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of hemels created on 25/03/2020, modeling between 227 and 238 tokens.The stress value of the MDS solution for the cloud of models is 0.166.

The main split in the cloud of models is given by FOC-WIN along the vertical axis; each half has three main subgroups: one for FOC-POS:nav on the left or below, striped by PPMI, one for PPMI:weight + FOC-POS:all in the middle, split by SOC-WIN, and a longer one to the right with PPMI:no | PPMI:selection + FOC-POS:all, cut orthogonally by PPMI and SOC-WIN (Figure 42).

Figure 42. Clouds of models of 'hemels'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=hemels'> here</a>.

Figure 42. Clouds of models of ‘hemels’. Explore it here.

As seen in Figure 43, PPMI interacts with other parameters so that their impact lowers when PPMI:weight. To compare the effect of the strongest parameters, a selection of models with LENGTH:FOC + SOC-WIN:4 + SOC-POS:nav will be examined, initially discarding the PPMI:selection models.

Figure 43. Distances between models of 'hemels' that vary along only one parameter, colored by `PPMI`.

Figure 43. Distances between models of ‘hemels’ that vary along only one parameter, colored by PPMI.

PPMI:no models lose one or two tokens, while their weighted counterparts may lose 3 to 11. As would be expected, according to the distance matrix the loosest model is the most different to all, with distances between 0.39 and 0.74, followed by it’s FOC-WIN:5 counterpart, with distances between 0.45 and 0.62 (but the distance between them is 0.30; see Distance matrix 4). For PPMI:weight models, FOC-POS makes for a distance of 0.12. These values are lowe when PPMI:no is replaced by PPMI:selection.

Distance matrix 4. Distance matrix between some models of ‘hemels’
id FOC-POS FOC-WIN PPMI
1 nav 10_10 weight
2 nav 10_10 no
3 all 10_10 weight
4 all 10_10 no
5 nav 5_5 weight
6 nav 5_5 no
7 all 5_5 weight
8 all 5_5 no

The MDS solutions show a small number of outliers that push most of the cloud into a condense center for the loosest model and to a lesser degree for FOC-WIN:5 + FOC-POS:all + PPMI:no and FOC-WIN:10 + FOC-POS:nav + PPMI:no. Outliers in a model tend to be peripheric in the others, but not lost by stricter measures. “Outliers” discusses some of the most evident cases. Color coding reveals a decent split between senses, with a tighter cluster of hemels_2 ‘wonderful’ tokens in the PPMI:weight models.

The t-SNE models show a small separated cluster (mainly for PPMI:weight models) already from perplexity 5, and a mass of tiny groups that starts taking shape with higher perplexities, to form three or four relatively distinct clusters (or at least areas with their own denser center). The small separated cluster seems to gather tokens with a culinary topic. Another tiny cluster that stands out in these models is made of 5 tokens of “hemels geschenk” ‘heavenly gift’, which is a fixed expression referring to something so good that seems to be a gift from Heaven.

Color coding lets us identify a small cluster of hemels_1 ‘divine’ tokens across all models in perplexities 20 and 30: these tokens are characterized by the presence of aards ‘earthly, mundane’, in these contexts the antonym of the target. The rest of the hemels_1 ‘divine’ tokens seem scattered among the hemels_2 ‘wonderful’ ones, but in FOC-WIN:5 models (except maybe FOC-POS:all + PPMI:no) this cluster seems to be at the core of a predominantly hemels_1 ‘divine’ subcloud, against the subcloud of hemels_2 ‘wonderful’ with tokens with a musical context at its core (Figure 44).

Figure 44. Tokens of 'hemels' in the t-SNE solutions (perplexity 30) of the selected models

Figure 44. Tokens of ‘hemels’ in the t-SNE solutions (perplexity 30) of the selected models

The rest of the parameters don’t seem to make much of a difference, but visually LENGTH:FOC seems to give nicer separability than LENGHT:5000 and after that choice SOC-WIN:10 better than SOC-WIN:5. An example of nice separability is Figure 45, with FOC-WIN:5 + FOC-POS:nav + PPMI:weight for the first order level and SOC-WIN:10 + SOC-POS:nav + LENGTH:FOC for the second.

Figure 45. Beautiful token cloud of 'hemels'.

Figure 45. Beautiful token cloud of ‘hemels’.

To look further into the clouds of ‘hemels’ it would be interesting to look into the cloud shown in Figure 45 and models that vary along one parameter from it, particularly its PPMI:selection and FOC-WIN:10 counterparts.

Outliers

There are two tokens that stand out as outliers of the PPMI:no + FOC-POS:all models (examples (74 and (75)) and three in PPMI:weight models (although to a lesser degree in FOC-WIN:5 + FOC-POS:nav): (76) through (78).

  1. vertrouwen op hun eigen kunnen , en zijn nu eenmaal mens … Hemelse haven , ons beloofd door een Vader , Zoon en Vriend ,
  2. Hemelse aardappelen Wie zin heeft in een letterlijk hemelse smaak , kan voortaan ruimte-aardappelen kopen
  3. de sterren2.101 zijn samen het0.099 volmaakte getal2.414 van hun cijfers , de hemelse nul1.607 van de tijd . Hamburg IJs , vuil ,
  4. geven0.309 tot0.107 de piste , en dan0.179 rechtuit rechtaan te spurten . Ik had hemelse schrik2.264 van Hammond . Nadien was Roger wel net zo0.197 blij als ik met0.259
  5. hoopt tijdens de strijd om deelname aan het0.099 wereldkampioenschap0.796 volgend jaar in Japan en Zuid-Korea op hemelse bijstand1.998 . Een0.329 geestelijke zal de voetbalschoenen van de Roemenen in een0.329 klooster bij

geldig

The adjective geldig was tagged with 2 definitions, reproduced in Table 25. It can be translated to English as “valid”, with a core, specific meaning in relation to laws and rules (geldig_1) and a broader one, related by perceived similarity (geldig_2). Unlike hemels, for this adjective the specific meaning is expected to be much more frequent than the general one, and no adverbial or ambiguous uses were reported in the pilot concordance.

Table 25. Definitions of ‘geldig’.
code definition example freq
geldig_1 van kracht, van toepassing, van waarde zijnde volgens wettelijke of andere regels een geldig vervoerbewijs, betaalmiddel, juridisch bewijs 32
geldig_2 van kracht, van toepassing, van waarde in ruimere zin een geldige redenering 8

Sense distribution

The sample consists of 240 tokens (6 batches) out of 5128 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 46. The distributions of the annotations (not majority senses) by annotator are shown in Figure 47. No batch was annotated by 4 annotators. The distribution across batches remains quite stable and very similar to the estimate, with only one token with no agreement.

Figure 46. Distribution of majority senses of 'geldig' per batch

Figure 46. Distribution of majority senses of ‘geldig’ per batch

Figure 47. Distribution of sense annotations of 'geldig' per annotator, grouped by batch.

Figure 47. Distribution of sense annotations of ‘geldig’ per annotator, grouped by batch.

“geldig” is an adjective with two senses related by perceived similarity: a core, specific one and a general one with much broader application, the former being much more frequent than the latter.

Confusion matrix

The confusion matrix between the majority senses and other tagged senses can be seen in Table 26 (raw number of tokens with such senses assigned) and Table 27 (mean confidence of such sense annotation in each token). Given the specificity of geldig_1 ‘valid-rules’ the confusion depends on the clarity of the context, but it’s expected to be low. The numbers don’t look that low, but in relative frequencies they are quite descent: 0.81 of the geldig_1 ‘valid-rules’ tokens and 0.67 of the geldig_2 ‘valid-general’ ones have full agreement and with high confidence: the alternative annotations tend to have lower confidence.

Table 26. Non weighted sense matrix of ‘geldig’ senses. Proportion of tokens with full agreement per sense-tag is: geldig_1: 0.81, geldig_2: 0.67.
senses geldig_1 geldig_2 unclear
geldig_1 203 38 0
geldig_2 12 36 0
no_agreement 1 1 1
total 216 75 1
Table 27. Weighted sense matrix of ‘geldig’ senses. Mean confidence across the lemma is 4.38; values above are darker and boldened. Median confidence across the lemma is 5.
senses geldig_1 geldig_2 unclear
geldig_1 4.51 3.47 0
geldig_2 3.92 4 0
no_agreement 2 5 0

There is one token where the annotators could not agree: (79). One assigned geldig_1 ‘valid-rules’, but with medium confidence, and selected as relevant context word Montesquieu (R3). Another one chose geldig_2 ‘valid-general’ with maximum confidence, and selected the following words as relevant: universeel (L0), onveranderlijk (L2), dus (L3), was (L4), aard (L5), en (L7). The third one assigned geen ‘none of the above’ with minimum confidence and no relevant context words and commented that there was too little context to base the choice on.

  1. aan de natuur of aan het opperwezen , en hun aard was dus onveranderlijk en universeel geldig . Montesquieu draait dat om : het is de wil van de bevolking die
    to the nature and the supreme being, and their nature was the unchangeably and universally valid. Montesquieu turns that around: it is the will of the people that

Nephology of geldig

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of geldig created on 25/03/2020, modeling between 235 and 239 tokens.The stress value of the MDS solution for the cloud of models is 0.183.

The main split of the cloud of models is given by FOC-WIN along the vertical dimension. Each half has three main subclouds, relatively disperse and close to each other compared to other clouds of models. The leftmost subcloud is made of PPMI:weight models, split by FOC-POS; the other two clouds are characterized by a FOC-POS and split by the other two PPMI values, with the FOC-POS:nav cloud closer to the PPMI:weight one than to FOC-POS:all (Figure 48).

Figure 48. Clouds of models of 'geldig'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=geldig'> here</a>.

Figure 48. Clouds of models of ‘geldig’. Explore it here.

The other three parameters make smaller groupings within the major ones, but Figure 49 shows that only LENGTH ever makes a difference greater than 0.2 (and FOC-WIN only does without PPMI:weight). To look into the effect of the stronger parameters, the classic LENGTH:FOC + SOC-WIN:4 + SOC-POS:nav will be selected, initially disregarding PPMI:selection.

Figure 49. Distances between models of 'geldig' that vary along only one parameter, colored by `PPMI`.

Figure 49. Distances between models of ‘geldig’ that vary along only one parameter, colored by PPMI.

Few tokens are lost by the selected models: one by FOC-POS:nav models, one more by PPMI:weight models and 2 by FOC-WIN:5 models, without overlap. Both PPMI:no + FOC-POS:all models stand out in the distance matrix with the highest distance values, between 0.33 and 0.65 (Distance matrix 5). The differences are less drastic with PPMI:no replaced by PPMI:selection.

Distance matrix 5. Distance matrix between some models of ‘geldig’
id FOC-POS FOC-WIN PPMI
1 nav 10_10 weight
2 nav 10_10 no
3 all 10_10 weight
4 all 10_10 no
5 nav 5_5 weight
6 nav 5_5 no
7 all 5_5 weight
8 all 5_5 no

When PPMI:no and PPMI:selection models are compared, it is not the loosest model that stands out. The difference between models varying only across PPMI is minimal (around 0.08 for FOC-POS:nav models and between 0.15 and 0.18 for FOC-POS:all), while the larger ones lie between models that vary across both FOC-POS and FOC-WIN (between 0.49 and 0.57, see Distance matrix 6).

Distance matrix 6. Distance matrix between some models of ‘geldig’
id FOC-POS FOC-WIN PPMI
1 nav 10_10 selection
2 nav 10_10 no
3 all 10_10 selection
4 all 10_10 no
5 nav 5_5 selection
6 nav 5_5 no
7 all 5_5 selection
8 all 5_5 no

The MDS solutions look more compact with PPMI:no and more spread with PPMI:weight, very rarely with outliers. The only outstanding outlier, in the loosest model and without that much power, is the token lost by PPMI:nav models, where the target occurs in a very short sentece: “Niet geldig?” ‘Not valid?’. Models that go beyond sentence boundaries will likely give different results. Color coding shows that the tokens of the minority sense, geldig_2 ‘valid-general’, do tend to stick together, with the majority of them grouped in the same quadrant and some others spread around. PPMI:weight models also seem to have some smaller clusters of geldig_1 ‘valid-rules’ within the greater cloud.

The t-SNE solutions look like an archipelago at perplexity 5 and rather uniform masses at perplexity 50. Perplexity 20 gives 5 or 6 relatively separated clusters in PPMI:weight and a uniform mass with three outer clusters in PPMI:no. One of the tightest and more clear clusters stands out in all models: it is made of tokens with a sports topic, mostly with the expression “een geldig doelpunt afkeuren” ‘rule out a valid goal’ (and one token with “goal niet valideren”!) but also other unrelated sport situations. Perplexity 30 seems to tighten the clusters in PPMI:no models and spread and separate those in PPMI:weight.

Color coding shows that the geldig_2 ‘valid-general’ tokens tend to cram together with perplexity 5 or 20 but are much more disperse with higher perplexities. It does make sense considering that it’s a minority sense and that the nouns it co-occurs with are not expected to belong to a clear category as is the case with the ones of the specific sense. The PPMI:selection models look a bit more structured than their PPMI:no counterparts; FOC-WIN:10 models do seem to have a better separability, but it could be interested to look further into models with different FOC-POS (Figure 50).

Figure 50. Tokens of 'lemma' in the t-SNE solutions (perplexity 30) of the selected models

Figure 50. Tokens of ‘lemma’ in the t-SNE solutions (perplexity 30) of the selected models

Second order parameters make very small differences in the distance matrix. Visually, LENGTH:FOC seems to give better separability to the clusters than LENGTH:5000, at least for FOC-WIN:10 models with a t-SNE solution.

For further inspection of “geldig” it would be interesting to look at FOC-WIN:10 + PPMI:weight models, maybe varying along the first order parameters but keeping SOC-WIN:4 + SOC-POS:nav and LENGTH:FOC fixed (since they don’t make much of a difference).

gemeen

The adjective gemeen was tagged with 5 definitions, reproduced in Table 28. It’s basic sense can be translated to “common, shared” (gemeen_1), with a similarity extension to “public” (gemeen_2) and to “common, average” (gemeen_3), which is extended to “ordinary, vulgar” (gemeen_5) and eventually to “mean, malicious” (gemeen_4). In the pilot concordance, the tentative tagging did not use these senses, so some of them were not counted as such. The great majority belongs to gemeen_4 ‘mean’ but includes malicious people, malicious actions and serious situations.

Table 28. Definitions of ‘gemeen’.
code definition example freq
gemeen_1 gemeenschappelijk in gebruik of bezit, gedeeld gemene kosten, een gemene muur 14
gemeen_2 openbaar, publiek de gemene zaak 2
gemeen_3 alledaags, gewoon, tot de middelmaat behorend het gemene volk, de gemene man 0
gemeen_4 boosaardig, kwaadaardig, laaghartig, malicieus een gemene streek 20
gemeen_5 ordinair, plat, onkies, vulgair gemene praatjes 0

Sense distribution

The sample consists of 320 tokens (8 batches) out of 2997 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 51. The distributions of the annotations (not majority senses) by annotator are shown in Figure 52. Batch 8 was annotated by 4 annotators. The distribution across batches seems relatively stable, with a small but persitent presence of gemeen_2 ‘public’, gemeen_3 ‘average’ and gemeen_5 ‘vulgar’, a large majority of gemeen_4 ‘malicious’ tokens and a decent but variable number of gemeen_1 ‘shared’ tokens and of cases with no agreement. All things considered, that distribution does not differ drastically from what was expected: previously unattested (or rather “not as such annotated”) senses occur, replacing some tokens of gemeen_1 ‘shared’, but the general proportions are kept.

Figure 51. Distribution of majority senses of 'gemeen' per batch

Figure 51. Distribution of majority senses of ‘gemeen’ per batch

Figure 52. Distribution of sense annotations of 'gemeen' per annotator, grouped by batch.

Figure 52. Distribution of sense annotations of ‘gemeen’ per annotator, grouped by batch.

“gemeen” is an adjective with multiple related senses: a core one that takes up about a tenth of the occurrences, three derived by similarity or metonymy and quite infrequent, and one with a weaker semantic link covering at least half the occurrences.

Confusion matrix

Matrices

The confusion matrix between the majority senses and other tagged senses can be seen in Table 29 (raw number of tokens with such senses assigned) and Table 30 (mean confidence of such sense annotation in each token). The distinction between the senses is rather subtle and they do not exclude each other, so ambiguous cases are to be expected. Confusion between gemeen_4 ‘malicious’ and others may come from cases where more than one sense may be present, while the difference between the other four senses are more subtle and depend strongly on how much information can be extracted from the context.

The confusion matrix shows indeed that gemeen_1 ‘shared’ and gemeen_4 ‘malicious’ present the least confusion, although there is quite a number of tokens with gemeen_4 ‘malicious’ (about a fourth of them) that received gemeen_5 ‘vulgar’ as alternative: that number is more than three times higher than the number of tokens that received gemeen_5 ‘vulgar’ as majority sense. This could suggest gemeen_5 ‘vulgar’ as a nuance that can be found mainly in tokens of gemeen_4 ‘malicious’ and ocasionally independently, or that these tokens simply have a predominantly pejorative meaning that some annotators connect more strongly with gemeen_5 ‘vulgar’ and others with gemeen_4 ‘malicious’. The minority and more subtle senses gemeen_2 ‘public’ and gemeen_5 ‘vulgar’ have the lowest rate of full agreement, with 0.19 for the former and 0 for the latter. For both and gemeen_3 ‘average’ the number of tokens that received these tags as alternatives is larger than those with them as majority senses.

Table 29. Non weighted sense matrix of ‘gemeen’ senses. Proportion of tokens with full agreement per sense-tag is: gemeen_1: 0.74, gemeen_2: 0.19, gemeen_3: 0.58, gemeen_4: 0.64, not_listed: 0.5.
senses gemeen_1 gemeen_2 gemeen_3 gemeen_4 gemeen_5 between not_listed unclear wrong_lemma
gemeen_1 72 7 10 1 0 0 0 1 0
gemeen_2 8 16 3 0 2 0 0 0 0
gemeen_3 1 0 12 2 1 0 1 0 0
gemeen_4 4 2 8 175 42 1 6 3 0
gemeen_5 0 2 2 7 13 0 0 2 0
not_listed 0 0 0 0 1 0 2 0 0
wrong_lemma 0 1 0 0 0 0 0 0 1
no_agreement 13 9 13 16 12 3 8 10 2
total 98 37 48 201 71 4 17 16 3

Regarding confidence, the agreeing gemeen_1 ‘shared’ tokens tend to have high values, but the rest are rather lower, with some mean values above 3 but mostly below. This makes sense considering the number of alternatives, the subtlety of the differences between them and that they are not really mutually exclusive.

Table 30. Weighted sense matrix of ‘gemeen’ senses. Mean confidence across the lemma is 3.54; values above are darker and boldened. Median confidence across the lemma is 4.
senses gemeen_1 gemeen_2 gemeen_3 gemeen_4 gemeen_5 between not_listed unclear wrong_lemma
gemeen_1 4.01 3.29 3.3 3 0 0 0 0 0
gemeen_2 3.25 3.1 2 0 2 0 0 0 0
gemeen_3 4 0 3.74 4.5 0 0 0 0 0
gemeen_4 3.25 0.5 2.38 3.75 2.9 2 2.5 2 0
gemeen_5 0 3 2.5 2.71 2.46 0 0 0 0
not_listed 0 0 0 0 3 0 1.75 0 0
wrong_lemma 0 3 0 0 0 0 0 0 3.5
no_agreement 2.69 3.39 2.38 2.72 1.92 2 2.5 1.85 3.5

Some tokens with geen ‘non of the above’ as majority sense or no agreement are discussed in their respective subsections.

geen as majority sense

There were three tokens that received a geen ‘none of the above’ tag with the same kind of explanation as majority sense. (80) was confusing for two annotations, so that one suggested it was the name of the “Kampen” (when that is the name of a town) and the other suggested it was the name of a place based on the capital letter. The main problem here is that (for an unknown reason) the word has its own separate sentence (which removes it from the current models) but the punctuation is not rendered in the text, so that it looks like part of the name. In the cases of (81) and (82), the annotators suggested a different sense. For the former, they suggested “hard/zward” ‘hard, tough’, not realizing that it was an adverbial usage7. For the latter, instead, one annotator assiged gemeen_5 ‘vulgar’, one suggested "nijpend’ ‘pressing’?? and the other one stated that it looked like a Netherlandic Dutch usage that expresses appreciation for the object modified by the target (the source is indeed from The Netherlands).

  1. veel sterkte en ook president Bush . Groep 8 van de Willem Alexanderschool in Kampen Gemeen Vanochtend ( 12 september ) zaten wij te werken . Ik was de boeken aan
    a lot of strength and also president Bush. Group 8 from the Willem Alexander school in Kampen. Public. This morning (September 12) we were working. I was […] the books
  2. Grunberg rond de wereld eventjes denkt te kunnen verorberen als een reuzenkoek , zal zich spoedig gemeen verslikken . ’ Als je jong bent , denk je dat er normale mensen
    Grunberg thinks to be able to devour something around the world like it is a giant cookie, will quickly choke on it hard. If you are young, you thinkt that normal people
  3. voltrekt zich het wonder . Opeens zijn daar die betoverende galmrijke gitaarlijnen , dat gemene bluesy slidespel en dat razendvirtuoze getokkel . In kamerbreed stereo natuurlijk , want die
    the wonder takes place. Suddenly there are those enchanting reverberating guitar lines?, that evil/pressing/awesome bluesy slide game and that infuriating strumming. In a wall to wall stereo of course, because those

No agreement

There is a high number of tokens with no agreement between the annotators, which is to be expected given the high number of possible annotations. Some of them were indeed quite ambiguous, although in some cases the suggestions were not really so reasonable. Some interesting cases are discussed below.

Sometimes, there is a [possible] pejorative nuance in the usage of the target, but which of the pejorative senses is meant is not so clear to the annotators. This could reflect how they interpret this kind of texts rather than how they understand the definitions. One such example is (83), to which one of the annotators assigend gemeen_3 ‘average’ with medium confidence, another one gemeen_4 ‘malicious’ with high confidence and a sense one geen ‘none of the above’ with minimum confidence and reporting hesitation between gemeen_4 ‘malicious’ and gemeen_5 ‘vulgar’.

  1. komt . Je herkent de boer , knecht , jack , valet aan zijn gemene kop , de vrouw , dame , koningin aan haar liefelijk gelaat en de heer of
    comes. You recognize the farmer, servant, jack, valet from his average/?malicious/vulgar head, the woman, dame, queen from her lovely face and the lord of

Some tokens are occurrences of the fixed expression “gemene zaak maken” ‘collaborate, lit. make common/shared business’, like in (84). This tokens is one of three that also belonged to the same batch (a few more were from other batches): one of the annotators assigned gemeen_1 ‘shared’ with medium confidence and identified the fixed expression in the comments section, another one suggested gemeen_2 ‘public’ with maximum confidence and the third one, geen ‘none of the above’ with high confidence, identifying and paraphrasing the expression in the comments section, so that it was classified as not_listed. It must be noted that each of these annotators assigned a higher number of gemeen_1 ‘shared’, gemeen_2 ‘public’ and not_listed respectively, in comparison to the other two of the same batch.

  1. overwinning . Veel Spanjaarden vinden dat Aznar de terreur over Spanje had afgeroepen door gemene zaak te maken met de Amerikanen in Irak . Polen boos Een
    victory. Many Spanish think that Aznar spread terror in Spain by collaborating (lit. making shared business) with the Americans and Irak. Poland angry A

Finally, two of the instances with no agreement turn out to be non adjectival uses of the target: adverbial in the case of (85) and nominal in the case of (86). In each case, one of the annotators remarked that it was the wrong part of speech, one just declared insufficient context and the other(s) suggested a sense label. In (85), the annotation that reported the wrong part of speech had medium confidence and the other two, suggesting gemeen_1 ‘shared’ and gemeen_5 ‘vulgar’ respectively, had minimum cofnidence. (86), on the other hand, was annotated by four annotators: the one that identified the wrong part of speech reported maximum confidence, explained what part of speech was being used and connected it to the meaning of “alledaags, gewoon” ‘everyday, common’ (gemeen_3?); another one just reported insufficient context, with minimum confidence, ant the other two assigned gemeen_3 ‘average’ with maximum confidenc and gemeen_5 ‘vulgar’ with medium confidence, respectively.

  1. uitzicht op vooruitgang , en zwoeren zelf ook te gaan koersen . Net zo gemeen hard als Van Hauwaert , de nieuwe rijke die nog niet zo lang daarvoor het geld
    view of progress, and even swore to go racing. Just as intensely? tough as Van Hauwert, the new rich person that not so long before […] the money
  2. genietingen van de hemelingen konden dan ook niet grondig genoeg verschillen van het gezwijn waarin het gemeen zijn genot zocht . Hier geen dieren , planten of rivieren in de hemel
    enjoyments of the heavens couldn’t in any case differ strongly enough from the ?? where the common (people) seeked their pleasure. No animals, plants or rivers in heaven

Nephology of gemeen

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of gemeen created on 25/03/2020, modeling between 314 and 317 tokens.The stress value of the MDS solution for the cloud of models is 0.149.

The cloud of models has a star shape with two small arms in the bottom left quadrant and four larger ones, and the main split is made by FOC-POS. The two small arms are PPMI:weight clouds, each a different FOC-POS and further split by FOC-WIN. The other four arms are spread as follows: one in the upper left and one above along close to the central vertical axis (y > 0, x ≈ 0), with FOC-POS:nav, each a different FOC-WIN and further split by PPMI, and then one to the right close to the central horizontal axis (y ≈ 0, x > 0) and one in the bottom right, with FOC-POS:all, each a different FOC-WIN and further split by PPMI (Figure 53).

Figure 53. Clouds of models of 'gemeen'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=gemeen'> here</a>.

Figure 53. Clouds of models of ‘gemeen’. Explore it here.

As shown in Figure 54, the distances between the models are rather large, but with PPMI:weight all other parameters make singlehandedly a rather small difference. To look at the effect of the strongest parameters, the weaker ones will be kept constant at LENGTH:FOC + SOC-WIN:4 + SOC-POS:nav, initially disregarding PPMI:selection.

Figure 54. Distances between models of 'gemeen' that vary along only one parameter, colored by `PPMI`.

Figure 54. Distances between models of ‘gemeen’ that vary along only one parameter, colored by PPMI.

Only one token is lost by FOC-POS:nav models, two in combination with FOC-WIN:5 and one more with all three restrictions at work. The distance matrix shows that the loosest model is the most different to the rest, followed by two other PPMI:no models: FOC-WIN:5 + FOC-POS:all and FOC-WIN:10 + FOC-POS:nav (Distance matrix 7). The difference is only slightly less drastic when PPMI:no is replaced by PPMI:selection; when comparing PPMI:no and PPMI:selection models, on the other hand, PPMI is the less impactful parameter.

Distance matrix 7. Distance matrix between some models of ‘gemeen’
id FOC-POS FOC-WIN PPMI
1 nav 10_10 weight
2 nav 10_10 no
3 all 10_10 weight
4 all 10_10 no
5 nav 5_5 weight
6 nav 5_5 no
7 all 5_5 weight
8 all 5_5 no

The MDS solutions are mainly distinguished by the presence of a tight flat subcloud in PPMI:weight models that stays separate from the main cloud (in some cases it een looks like a great ship moored at a bay, since the bigger cloud arches around it without touching it). These tokens also seem to stick together in PPMI:no models, more or less tightly, but without so much distance between their subcloud and the main one. This is a cluster of “(grootste) gemene deler” ‘(greatest) common divisor’ and covers most of the instances of gemeen_1 ‘shared’. It does not overlap with the large gemeen_4 ‘malicious’ cloud, but other than that there are no sense-specific areas to speak of.

The t-SNE solutions are also strongly marked by the “gemene deler” ‘common divisor’ cluster: it already stands out from the rest of the tiny cluster with perplexity 5, in all models but more clearly in PPMI:weight | FOC-WIN:5, and becomes tighter and more independent (farther from the rest) as the perplexity increases, again more clearly in FOC-WIN:5 models and even more in PPMI:weight models. This groups the rest of the tokens together in a rather uniform cloud in which little structure can be seen, with or without color coding. The gemeen_4 ‘malicious’ tokens predominate out of sheer frequency and no inner groupings can be identified at this stage. To a certain degree, PPMI:selection models show a bit more structure (see Figure 55), particularly with FOC-WIN:10 + FOC-POS:nav by at least grouping most of the remaining gemeen_1 ‘shared’ tokens (the core of this cluster is made of tokens with the expression “iets gemeen hebben met iemand” ‘have something in common with someone’), more with LENGTH:5000 than with LENGTH:FOC (Figure 56, with first order features set to FOC-WIN:5 + FOC-POS:nav + PPMI:selection and second order to SOC-WIN:4 + SOC-POS:nav). The difference made by other second order parameters is neglectable.

Figure 57. Tokens of 'gemeen' in the t-SNE solutions (perplexity 30) of the selected models

Figure 57. Tokens of ‘gemeen’ in the t-SNE solutions (perplexity 30) of the selected models

Figure 56. Nicest clouds of 'gemeen'.

Figure 56. Nicest clouds of ‘gemeen’.

For further inspection of ‘gemeen’ it could be interesting to look into the clouds in Figure 56 and maybe variations along one parameter.

goedkoop

The adjective goedkoop was tagged with 4 definitions, reproduced in Table 31. It basically translates to “cheap”, with the senses linked to the respective application to cheap goods (goedkoop_1), a shop or shopkeeper that sells cheep goods (goedkoop_2) and a place or area where prices tend to be low (goedkoop_3), and a metaphoric extension to banal or superficial abstract things (goedkoop_4). The first sense is expected to cover at least half the occurrences and goedkoop_3 ‘cheap-area’ to be the least frequent. There was also a number of ambiguous or confusin cases in the pilot annotation.

Table 31. Definitions of ‘goedkoop’.
code definition example freq
goedkoop_1 laag in prijs, betaalbaar, voordelig goedkope wijn 22
goedkoop_2 geen hoge prijzen vragend een goedkoop winkeltje, een goedkope loodgieter 7
goedkoop_3 waar de prijzen laag zijn een goedkope buurt 1
goedkoop_4 van weinig waarde, makkelijk verkregen, oppervlakkig, banaal goedkope lof, goedkoop succes, goedkope argumenten 4

Sense distribution

The sample consists of 320 tokens (8 batches) out of 40669 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 58. The distributions of the annotations (not majority senses) by annotator are shown in Figure 59. Batch 3 was annotated by 4 annotators. The distribution across batches looks less stable than in other lemmas; goedkoop_1 ‘cheap-goods’ is consistently the most frequent one and goedkoop_3 ‘cheap-area’ occurs only in half the batches, and once in each, while the frequencies of goedkoop_2 ‘cheap-seller’ and goedkoop_4 ‘banal’ vary widely from batch to batch and end up equal in the overall distribution. There is also a relatively high number of tokens with no agreement and none with geen ‘none of the above’ as majority sense: if we take this into the goedkoop_1 ‘cheap-goods’ group, the overall and estimate distributions are extremely similar.
Figure 58. Distribution of majority senses of 'goedkoop' per batch

Figure 58. Distribution of majority senses of ‘goedkoop’ per batch

Figure 59. Distribution of sense annotations of 'goedkoop' per annotator, grouped by batch.

Figure 59. Distribution of sense annotations of ‘goedkoop’ per annotator, grouped by batch.

“goedkoop” is an adjective with at least three senses, a central one that covers 50 to 75% of the occurrences, a metonymical extension and a metaphorical extension, these with similar frequencies.

Confusion matrix

Matrices

The confusion matrix between the majority senses and other tagged senses can be seen in Table 32 (raw number of tokens with such senses assigned) and Table 33 (mean confidence of such sense annotation in each token). The distinction between the senses is rather subtle so some confusion is to be expected, more between goedkoop_1 ‘cheap-goods’ and its metonymic extension goedkoop_2 ‘cheap-seller’ than between any of them and the metaphoric goedkoop_4 ‘banal’. That is indeed the case, with two thirds of the tokens agreeing on goedkoop_1 ‘cheap-goods’ and of goedkoop_4 ‘banal’ with full agreement and most confusion between the first three senses than between goedkoop_4 ‘banal’ and others: there are more annotations of goedkoop_2 ‘cheap-seller’ and goedkoop_3 ‘cheap-place’ as alternatives than as majority sense. This probably come mostly from one annotator (annotator 3 in batches 2 and 6) with a clear tendency to assign these two tags over the other annotators in these batches.

According to the weighted matrix, confidence values are also highest for the agreeing goedkoop_1 ‘cheap-goods’ and goedkoop_4 ‘banal’, but also to all goedkoop_1 ‘cheap-goods’ annotations except for those on tokens with goedkoop_4 ‘banal’ as majority sense.

Table 32. Non weighted sense matrix of ‘goedkoop’ senses. Proportion of tokens with full agreement per sense-tag is: goedkoop_1: 0.67, goedkoop_2: 0.31, goedkoop_3: 0.6, goedkoop_4: 0.66.
senses goedkoop_1 goedkoop_2 goedkoop_3 goedkoop_4 between not_listed unclear
goedkoop_1 237 32 23 18 1 0 5
goedkoop_2 16 29 2 3 0 0 0
goedkoop_3 2 0 5 0 0 0 0
goedkoop_4 7 1 1 29 0 1 0
unclear 1 0 0 1 0 0 1
no_agreement 18 11 11 10 0 0 1
total 281 73 42 61 1 1 7
Table 33. Weighted sense matrix of ‘goedkoop’ senses. Mean confidence across the lemma is 3.92; values above are darker and boldened. Median confidence across the lemma is 4.
senses goedkoop_1 goedkoop_2 goedkoop_3 goedkoop_4 between not_listed unclear
goedkoop_1 4.08 2.56 2.91 2.94 3 0 1
goedkoop_2 4.25 3.75 3.5 2.67 0 0 0
goedkoop_3 4.5 0 3.77 0 0 0 0
goedkoop_4 3.29 2 4 4.01 0 3 0
unclear 5 0 0 3 0 0 0
no_agreement 4.42 2.91 2.86 3.55 0 0 1

Tokens with goedkoop_1 ‘cheap-goods’ and goedkoop_4 ‘banal’ as majority sense but no full agreement and those with no agreement are discussed in their respective sections.

Mostly metaphoric

Out of the 29 tokens with goedkoop_4 ‘banal’ as majority sense, 10 received an alternative annotation, 7 of which were goedkoop_1 ‘cheap-goods’. Not all of them will be described here, but some of them proved to be rather interesting.

Of those with goedkoop_1 ‘cheap-goods’ as alternative, two were actually adverbial uses, like in (87), two are actually a very clear goedkoop_1 ‘cheap-goods’ case, like (88) and one with “gelijkmaker” ‘equalizer’, (89) shows an interesting challenge, one is definitely goedkoop_5 ‘banal’ and one has a valid double sense, with “stationsromannetje”, the Belgian name for an ‘airport novel’.

  1. nu aan Adriaanse de opstelling voor de wedstrijd tegen Celtic doorgeeft . Dit is goedkoop scoren . " Want waarom zouden vorstelijk gehonoreerde voetballers per definitie als slachtoffers moeten
    now that it passes the setup of the match against Celtic to Adriaanse. This is to score cheaply. " Because why should royally honored football players […] by the definition as victims
  2. Hans Van Haelemeesch , communicatiemanager van de touroperator Jetair : " Of de goedkope dollar een effect heeft op het huidige zomerseizoen , kunnen we nu niet meer afmeten :
    Hans Van Haelemesch, communication manager of the Jetair tour operator: " Whether the cheap dollar will have an effect on the current summer season, we can’t estimate yet:

When annotating (89), the annotators that chose goedkoop_4 ‘banal’ selected rommel (R0) as relevant context word, while the one that assigned goedkoop_1 ‘cheap-goods’ selected 200 (R10), frank (R11), per (R12), stuk (R13).

  1. en de steun van Canon Cultuurcel . Maar we wilden degelijk materiaal en geen goedkope rommel aanbieden . De prenten alleen al kostten 200 frank per stuk ( op
    and the support of Canon Culture Cell. But we wanted to offer solid material and no cheap garbage. Printing alone cost already 200 franks per piece (on

The token that received goedkoop_2 ‘cheap-seller’ as alternative was a very clear case of goedkoop_4 ‘banal’, but the one receiving goedkoop_3 ‘cheap-place’ (90) actually matches one of the examples of goedkoop_2 ‘cheap-seller’ (although it could be mixed with goedkoop_3 ‘cheap-place’, since the object is “winkel” ‘store’). Maybe the context suggests something in the lines of goedkoop_4 ‘banal’ that I don’t see?

  1. te vervullen . Maar aan het einde van de basisschool doen kinderen zo’n ` goedkope ’ winkel in de ban . Dan veranderen ze opeens in merkensnobs , zo
    to fulfill. But at the end of elementary school the children ban such a ‘cheap’ store. Later they suddenly turn into brand snobs, so

Finally, the example with not_listed as alternative is also interesting, in that this concordance (91) shows a fixed expression where the meaning of the target is very close to that of goedkoop_4 ‘banal’, but in any case the part of speech is adverbial.

  1. huivert hij nog na . " Ik kwam er met twee gekneusde knieën nog goedkoop vanaf . De toestand van mijn vriendin baarde me meer zorgen .
    he shuddered. "With two bruised knees I got off ‘cheaply’. The condition of my girlfriend worried me more.

Mostly goods

There are 18 tokens with gemeen_1 ‘cheap-goods’ as majority sense and gemeen_4 ‘banal’ as alternative. The great majority of those examples just show subjectivities in the annotators. Cheap goods can often be regarded as having “little value”, which is one of the paraphrases of the goedkoop_4 ‘banal’ definition, but was meant to be applied to rather abstract entities; in many of these examples, the predominant sense is gemeen_1 ‘cheap-goods’ and there is no much in the text that points to a pejorative nuance. Four of these examples had four annotators, of which the same one consistently and with high confidence chose gemeen_4 ‘banal’. In the rest of the cases, the dissident annotation tends to have lower confidence.

One example, (92), could be considered more interesting, since the use of quotation marks suggests an ironic or at least stylistic usage (the disagreeing annotator does point out that it could have double sense and even be ironic).

  1. . Ruim driehonderd gulden . Dat zou helemaal opgaan aan de ’ goedkope ’ medicijnen . ’ Wat moeten we dan eten ? ’ , zegt Nomvula
    . About three hundred florins. All that should go to the ‘cheap’ medicines. ‘What do we have to eat then?’, says Nomvula

No agreement

There are 19 tokens of goedkoop where the annotators could not agree. In all but one of them, one of the annotations is goedkoop_1 ‘cheap-goods’, and that is normally the most adequate answer: the situation is simillar to that one described in “Mostly goods”. Sometimes the sold object is a place or region, in which case some annotators get confused between the appropriate goedkoop_1 ‘cheap-goods’, goedkoop_2 ‘cheap-seller’, of which one example was “goedkoop winkeltje” ‘cheap store’, and goedkoop_3 ‘cheap-area’. One such example is (93), where the goods to be bought are land.

  1. Let wel , dit zijn de marginale landbouwgebieden . Daar kan de gemeente nog goedkoop grond verwerven . Ik denk dat ze niet goed weten waarmee ze bezig zijn
    Be aware, these are marginal farming areas. Hier the town can still acquire cheap land. I think that they don’t know very well what they’re dealing with

The one instance where goedkoop_1 ‘cheap-goods’ was not a suggestion (94), but the other three were, was an adverbial use applied to “werken” ‘work’. Similar cases (“goedkoper produceren” ‘produce more cheaply’, “goedkoper worden georganiseerd” ‘be organized more cheaply’) received the first three senses as alternatives.

  1. bonden en Lisv . ’ Het was een voorwaarde voor de subsidie dat daardoor goedkoper gewerkt zou worden . Dat kost banen ’ , laat het GAK weten .
    unions and Lisv. ‘It was a condition for the subsidy that they would work more cheaply that way. That costs jobs.’, GAK tells.

Six of these 19 tokens were annotated by four annotators. In two of them they were split between goedkoop_1 ‘cheap-goods’ and goedkoop_4 ‘banal’, following the reasoning explained in “Mostly goods”; in three of them they were split between goedkoop_1 ‘cheap-goods’ and goedkoop_2 ‘cheap-seller’, but they are not really ambiguous cases. (95) fits well with goedkoop_1 ‘cheap-goods’ (one of the goedkoop_2 ‘cheap-seller’ annotators actually thought it may be either) and (96) with goedkoop_2 ‘cheap-seller’ (hopefully, unless the person is considered the goods, but somehow all annotations had maximum confidence except for one of the goedkoop_2 ‘cheap-seller’ ones). (97) is a separate case, similar to (98), the six token, where the annotators were split between goedkoop_1 ‘cheap-goods’ and goedkoop_3 ‘cheap-area’: they are adverbial uses that, in combination with uit, indicate that someone obtained something at a low price (without specifying what).

  1. . Bovendien is de productie van de deuren in een Chinese fabriek een stuk goedkoper . De voorbereidingen voor de Chinese fabriek hebben veel tijd en moeite gekost ,
    . Moreover the production of the doors in a Chinese factory is a bit cheaper. The preparations for the Chinese factory have cost a lot of time and effort,
  2. worden . ’ Alleen maar heel hard blijven roepen dat je de beste en goedkoopste bent , werkt niet meer ? ’ Een merk kan zijn problemen niet altijd
    become. ‘To just keep loudly announcing that you are the best and the cheapest, that doesn’t work anymore?’ A brand cannot always […] its problems
  3. ter waarde van 5000 euro via internet is de belegger - volgens Binck - 52 procent goedkoper uit dan bij ABN Amro . Een aandelentransactie bij de online broker kost 15
    with a value of 5000 euro via internet the investor gets off -according to Binck- 52 percent more cheaply than with ABN Amro. A shares transaction through the online broker costs 15.
  4. Ze kan het bovendien niet verkroppen dat stadsdeel Oud Zuid met de nieuwe parkeerbeheerders goedkoper uit is , terwijl zij voor het verlies van Stadstoezicht moet bloeden . Zijn
    Moreover she cannot digest that the Old South city district got off more cheaply with the new parking managers, while she had to bleed for the loss of City Surveillance. His

Nephology of goedkoop

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of goedkoop created on 25/03/2020, modeling between 314 and 320 tokens.The stress value of the MDS solution for the cloud of models is 0.158.

The main split is given by FOC-WIN along the vertical axis and a bit tilted. Each half can be further divided by PPMI and FOC-POS, with the leftmost group made up of all PPMI:weight models further split by FOC-POS, a middle section with FOC-POS:nav models further split by PPMI, and then two separates strips of FOC-POS:all models, each with a different PPMI. Furthermore, the SOC-WIN:10 models in the upper (FOC-WIN:5) half tend to go towards the center of the plot (Figure 60).

Figure 60. Clouds of models of 'goedkoop'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=goedkoop'> here</a>.

Figure 60. Clouds of models of ‘goedkoop’. Explore it here.

Figure 61 also shows that the parameters that make the most difference singlehandedly are FOC-POS if PPMI:weight, FOC-WIN and PPMI, at least with FOC-POS:nav. For further inspection, models with SOC-POS:nav + SOC-WIN:4 + LENGTH:FOC will be selected, initially discarding PPMI:selection.

Figure 61. Distances between models of 'goedkoop' that vary along only one parameter, colored by `PPMI` and `FOC-POS`.

Figure 61. Distances between models of ‘goedkoop’ that vary along only one parameter, colored by PPMI and FOC-POS.

One token is lost by PPMI:weight models, and 4-5 more with FOC-WIN:5. The distance matrix show the loosest model as the most different of all, with its smaller distances to other two PPMI:no models, namely FOC-WIN:10 + FOC-POS:nav and FOC-WIN:5 + FOC-POS:all, with 0.4 and 0.33 respectively. FOC-POS makes little difference between PPMI:weight models, as shown by Figure 61.

In the MDS solutions, the loosest model has a couple of outliers in lines where the target’s sentence is very short, but they are not too strong, and the other models don’t look too different from each other. Given the skewness of the sense distribution, the color coding is dominated by goedkoop_1 ‘cheap-goods’; the other senses don’t even cover specific areas. Everything overlaps.

The t-SNE solutions look like unstructured archipelagos with perplexity 5 and only some identifiable clusters in PPMI:weight and maybe FOC-POS:nav models with higher perplexity. PPMI:no models, especially with FOC-POS:all, quickly turn into uniform masses, maybe with a hint of dense cores. The most notable clusters are a big one put together by the co-occurrence with “zijn” ‘to be’ (which is to say the least unhelpful) and a small tight one of “goedkope arbeid(skrachten)” ‘cheap work (force)’. Color coding does not offer much more light (r); inspection of other possible clusters will be performed in later stages of the analysis.

Figure 62. Tokens of 'goedkoop' in the t-SNE solutions (perplexity 30) of the selected models

Figure 62. Tokens of ‘goedkoop’ in the t-SNE solutions (perplexity 30) of the selected models

PPMI:selection does not improve much upon PPMI:no, and the second order parameters make very little difference, so it should be safe to go to the default values.

For further inspection it would be interesting to examine the cloud with FOC-POS:nav + FOC-WIN:5 + PPMI:weight as first order parameters and SOC-POS:nav + SOC-WIN:4 + PPMI:no for the second order.

grijs

The adjective grijs was tagged with 6 definitions, reproduced in Table 34. It translates to “gray”, with sense distinctions referring to the actual visual color (grijs_1), specific instances such as cloudy times (grijs_2) and gray hair because of old age (grijs_3), with a further metonymic extension to old people (grijs_4), and then metaphoric extensions meaning “boring” (grijs_5) and “morally doubtful” (grijs_6). The basic meaning (grijs_1) was overwhelmingly frequent in the pilot concordance, covering half the occurrences; the rest were rather infrequent, at most around 10% of the concordance, and there was a relatively high number of tokens to which no sense could be assigned, half of which were instances of grijze middenmoot, which refers to the middle position in a ranking.

Table 34. Definitions of ‘grijs’.
code definition example freq
grijs_1 met een kleur die ligt tussen wit en zwart; vaalwit, grauw grijs van het stof, de grijze dolfijn 20
grijs_2 (van periodes e.d.) zonder veel zonneschijn, bewolkt, betrokken een grijze dag 4
grijs_3 (van haar) zijn kleur verloren hebbend, m.n. door gevorderde leeftijd een grijs baardje 0
grijs_4 (van personen e.a.) grijsharig, en vandaar, betrekking hebbend op ouderen de grijze golf 1
grijs_5 saai, kleurloos, vervelend een grijze buurt 4
grijs_6 niet helemaal volgens de wet of de regels, halflegaal de grijze economie 5

Sense distribution

The sample consists of 320 tokens (8 batches) out of 13567 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 63. The distributions of the annotations (not majority senses) by annotator are shown in Figure 64. Batch 4 was annotated by 4 annotators. All senses where attested in all batches, except for grijs_4 ‘old’ in batches 5 and 6. In general, the distribution is variable across batches, and the overall proportions are similar to the estimate but show a lower frequency of grijs_1 ‘gray-visual’, grijs_6 ‘gray-moral’ and geen ‘none of the above in favour of an actual presence of grijs_3 ’gray-hair’ and larger frequency of grijs_4 ‘old’. There is a number of tokens with low agreement.

Figure 63. Distribution of majority senses of 'grijs' per batch

Figure 63. Distribution of majority senses of ‘grijs’ per batch

Figure 64. Distribution of sense annotations of 'grijs' per annotator, grouped by batch.

Figure 64. Distribution of sense annotations of ‘grijs’ per annotator, grouped by batch.

“grijs” is an adjective with a frequent core visual meaning and a number of metonymic and metaphoric extensions of much lower but no neglectable frequency.

Confusion matrix

Matrices

The confusion matrix between the majority senses and other tagged senses can be seen in Table 35 (raw number of tokens with such senses assigned) and Table 36 (mean confidence of such sense annotation in each token). A certain confusion is expected between the core grijs_1 ‘gray-visual’ sense and the metonymic extensions, particularly grijs_2 ‘cloudy’ and grijs_3 ‘gray-hair’ (so that the tokens of the metonymical extensions may receive the core as alternative, rather than the other way around), and between grijs_3 ‘gray-hair’ and grijs_4 ‘old’; there could also be ambiguity between grijs_1 ‘gray-visual’ and grijs_5 ‘boring’ for certain entities, but little confusion is expected between the metaphoric grijs_6 ‘gray-moral’ and any of the others. The proportion of tokens with full agreement is exceptionally high for grijs_1 ‘gray-visual’, reaching 0.9, while there are 30 tokens when that was just the alternative annotation. It is also the sense with a wider variety of alternatives, covering all other options, while other senses get confused with subsets: there is never confusion between grijs_2 ‘cloudy’ on one hand and grijs_3 ‘gray-hair’ or grijs_4 ‘old’ on the other (a relief); grijs_3 ‘gray-hair’ is only confused with grijs_1 ‘gray-visual’ and grijs_4 ‘old’ and there is very little confusion between metaphoric senses on one side (grijs_5 ‘boring’ and grijs_6 ‘gray-moral’) and metonymic ones (grijs_2 ‘cloudy’, grijs_3 ‘gray-hair’, grijs_4 ‘old’) on the other. The number of not_listed cases is, on the other hand, quite remarkable, with 9 agreeing tokens and 22% full agreement (which is low for a sense, but high for this category); they are discussed in the “Alternative senses” subsection. Cases of no agreement seem to mostly involve grijs_5 ‘boring’ followed by grijs_1 ‘gray-visual’, grijs_6 ‘gray-moral’ and then not_listed; they are discussed in “No agreement”.

Table 35. Non weighted sense matrix of ‘grijs’ senses. Proportion of tokens with full agreement per sense-tag is: grijs_1: 0.9, grijs_2: 0.67, grijs_3: 0.76, grijs_4: 0.52, grijs_5: 0.49, grijs_6: 0.42, not_listed: 0.22.
senses grijs_1 grijs_2 grijs_3 grijs_4 grijs_5 grijs_6 between not_listed unclear
grijs_1 134 1 1 2 5 2 0 2 1
grijs_2 5 24 0 0 3 1 0 0 0
grijs_3 2 0 34 6 0 0 0 0 0
grijs_4 0 0 6 27 4 1 1 1 0
grijs_5 6 2 0 3 49 7 1 3 3
grijs_6 6 0 0 1 3 26 0 5 1
not_listed 2 0 0 0 3 1 0 9 1
unclear 1 0 0 0 0 0 0 0 1
no_agreement 8 3 1 3 11 8 1 7 1
total 164 30 42 42 78 46 3 27 8

Mean confidence values are highest for the agreeing annotations of grijs_1 ‘gray-visual’, grijs_2 ‘cloudy’ and grijs_3 ‘gray-hair’, but also to the (smaller in number) confused annotations between grijs_1 ‘gray-visual’ and grijs_5 ‘boring’ and between grijs_3 ‘gray-hair’ and grijs_4 ‘old’. The agreeing grijs_6 ‘gray-moral’ annotations, on the other hand, have a mean confidence below 3.

Table 36. Weighted sense matrix of ‘grijs’ senses. Mean confidence across the lemma is 3.97; values above are darker and boldened. Median confidence across the lemma is 5.
senses grijs_1 grijs_2 grijs_3 grijs_4 grijs_5 grijs_6 between not_listed unclear
grijs_1 4.4 5 5 3.5 4 2 0 2.5 1
grijs_2 3.4 4.34 0 0 3 0 0 0 0
grijs_3 3 0 4.65 4 0 0 0 0 0
grijs_4 0 0 4.5 3.91 3 1 2 1 0
grijs_5 4.17 3 0 3.67 3.38 3.29 2 3 1.67
grijs_6 2 0 0 2 2 2.9 0 3 1
not_listed 2 0 0 0 1.67 4 0 2.94 4
unclear 2 0 0 0 0 0 0 0 0.5
no_agreement 3.12 4 3.5 3.33 3 3.12 0 2.71 4

Further potentially interesting cases of confusion will be looked into later.

No agreement

16 tokens show no agreement between the annotators: in 5 of which there were four annotators split between two options. Most of them show confusion between grijs_5 ‘boring’, grijs_6 ‘gray-moral’ and something else, suggesting that a metaphoric interpretation was deemed necessary but the target domain is not clear. Particularly in examples (99) through (102), where the third alternative was an unlisted sense, are quite interesting. The respective suggestions are: “genuanceerder” ‘more nuanced’ for (99), “ver (in het verleden)” ‘far (in the past)’ for (100), “gewoon, niet speciaal” ‘common, not special’ for (101) and “de middenmoot” ‘middle of the ranking’ for (102).

  1. zijn symptomatisch voor de Duitse media en de politiek . De werkelijkheid is echter grijzer dan de meeste politici en commentatoren beweren . Vooral de constante herhaling van het
    are symptomatic for the German media and the politics. The reality is actually grayer than most politicians and commentators claim. Mostly the constant repetition of the
  2. en beseften dat Terpstra wist waarover ze sprak . De bewindsvrouw had in een grijs verleden immers zelf ook te maken gehad met de weerbarstige wetten der topsport .
    and realized that Terpstra knew what she talked about. After all in a gray past the minister herself had had something to do with the unruly laws of the top sport
  3. Toch denkt ruim driekwart van de Nederlanders dat groene stroom veel duurder is dan grijze stroom . Energiebedrijven moeten hun klanten nog veel duidelijk maken , zegt A. Goedmakers
    Yet about three quarters of the Dutch think that green energy is much more expensive than gray energy. Energy companies must still clarify a lot to their clients, says A. Goedmakers
  4. " GBA-trainer Brys : ’ Wie onze thuiswedstrijden gezien heeft , kan ons geen grijze ploeg noemen ’ Darco Pivaljevic bezorgde met de enige treffer van de avond Antwerp
    " GBA-trainer Brys: ‘Those who have seen hour home matches cannot call us a gray team’ Darco Pivaljevic delivered with the only hit of the evening Antwerp

There were also two instances of “grijze muis” ‘lit. gray mouse’, which refers to an unremarkable person, that were annotated as grijs_1 ‘gray-visual’ by two annotators (as expected, since it’s the whole expression that is metaphoric) and as grijs_5 ‘boring’ by the other two, which also makes sense. Another token with an equal split between four annotators and an actual instance of both senses simoultaneously is one of “grijze kenteken” ‘gray license plates’, referring to a category of (gray colored) license plates in The Netherlands that indicate a vehicle with tax benefits. The rest of the cases are not particularly relevant or revealing, except maybe for some misunderstandings of the definitions.

Alternative senses

9 tokens were assigned geen ‘none of the above’ with an explanation fitting the not_listed category by most of the annotators; the sense tags are quite consistent, so it would be appropriate to count it as a new, infrequent sense. (103) through (108) belong to the same batch, meaning they were annotated by the same three people.

In (103) through (106) the annotations are almost the same. The first annotator suggested a sense on the lines of “between [two things]” (“between good and bad” for (105)), always with minimum confidence (maybe because the tag was geen ‘none of the above’). The second one suggested “vaag, onduidelijk” ‘vague, unclear’ for all but (106), where they reported insufficient context, always with high confidence. Finally, the third one assigned grijs_5 ‘boring’ to (103) with low confidence and suggested “onduidelijke mix, zonder orde” ‘unclear mix, without order’ for (104) with high confidence and “gematigde zone tussen twee uitersten” ‘moderate zone between two extremes’ for the other two, with medium confidence and reporting insufficient context.

  1. naar de vroegere stijl . C & A lijkt zich hiermee weer in de grijze middenmarkt te storten . Deskundigen zetten vraagtekens bij de nieuwe strategie , daar ook
    to the previous style. C&A seems to put itself back in the gray middle market. Experts have questions about the new strategy, also
  2. ’ echt belangrijke zaken niet werden besproken . ’ " Het wordt nu een grijze brij van niet-heldere standpunten . " De PvdA wilde ’ duidelijkheid ’ , en
    ‘really important issues were not discussed.’ “Now it becomes a gray mash of unclear points of view.” The PvdA wanted ‘clarity’, and
  3. van de vragen goed beantwoord . Deze slechtste school is niet zwart , maar grijs . Op de school met de hoogste scores wisten de leerlingen gemiddeld het antwoord
    answered well […] of the questions. This worst school is not black, but gray. At the school with the highest scores the students knew in average the answer
  4. . De jaren negentig duren voort op de Balken . Maar nu grijs in plaats van zwartwit .
    . The nineties go on at the Balken. But now gray instead of black and white.

Examples (107) and (108) all received the same suggestion from all three annotators: it’s a fixed expression “een [muziek]nummer grijs draaien” ‘turn a [music] number gray’, meaning to play a song extremely often (one annotator included the nuance that it becomes boring as a consequence). The first annotator assigned low confidence and the other two, high.

  1. gezien ; tijdens Pinkpop . Een cassettebandje van hun muziek draaide ik daarna helemaal grijs . De mooiste scène is voor mij die wanneer The Happy Mondays totaal onvoorbereid
    seen; during Pinkpop. Afterwards I wore a casette tape of their music out completely (lit. I turned a cassette tape completely gray). The most beautiful scene for me is that one when The Happy Mondays […] totally unprepared
  2. jongens al een behoorlijke bekendheid gekregen . Regionale en lokale zenders draaien het nummer grijs en zelfs megaconcern Pepsi-Cola koos het nummer voor hun wintercommercial op televisie . Ook
    boys already received a decent fame. Regional and local stations play the number endlessly (lit. turn the number gray) and even the industry giant Pepsi-Cola chose the number for their winter commercial on TV. Also

The concordance reproduced in (109) is very similar to (104), but they come from different sources (different lands and different years). The annotators of (109) have different suggestions to the ones from (104). One assigned grijs_6 ‘gray-moral’; another one explained that it rather seemed to mean that there were no more clear boundaries, and that words in the surroundings suggested grijs_6 ‘gray-moral’ but it didn’t really feel right; the last one said it was a metaphor from grijs_1 ‘gray-visual’, based on the fact that mixing all colors gives gray and the different colors cannot be distinguished anymore.

  1. De traditionele partijen kunnen maar beter bang zijn . Het is allemaal één grijze brij geworden : CD&V heeft geen profiel , de SP.A en de VLD groeien naar mekaar
    The traditional parties should be scared. It has become one gray mash: CD&V has no profile, the SP.A and the VLD grow closely to each other

Examples (110) and (111) come from the same batch and exemplify the same concept, namely ‘gray matter’. In each case, one of the annotators assigned grijs_1 ‘gray-visual’ with medium confidence and the other two, with high confidence except in one annotation, clarified that the brain cells are not literally gray, but that is a name to signal a kind of brain mass. One of them also added that the ultimate meaning of the expression is “het verstand” ‘the mind’.

  1. . " Een verwittigd man is er twee waard en dus tilden we onze grijze hersencellen uit een situatie van seizoensgebonden werkeloosheid . In een vlaag van zinsverbijstering begonnen
    . "and informed man is worth two and therefore we lifted our gray matter (lit. braincells) from a situation of seasonal unemployement. In a fit of sensorial bewilderment […] started
  2. Nacht van het Examen ’ ANTWERPEN Op 24 maart kunnen studenten hun grijze hersenmassa weer testen op de vijfde Nacht van het Examen . Met de vergaarde
    Night of the exam ’ANTWERPEN On March 24 the students can test their gray [brain]matter again on the fifth Night of the Exam. With the amassed

Nephology of grijs

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of grijs created on 25/03/2020, modeling between 304 and 319 tokens.The stress value of the MDS solution for the cloud of models is 0.176.

The main split in the cloud of models is given by PPMI:weight | FOC-POS:nav on one side and FOC-POS:all with the rest of PPMI on the other side, along the horizontal axis. This is different to all other previous clouds. Each half is split by PPMI (the PPMI:weight section is split by FOC-POS) and then by FOC-WIN, both along the same direction. The FOC-POS:all clouds are split by SOC-WIN more clearly than by FOC-WIN (Figure 65).

Figure 65. Clouds of models of 'grijs'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=grijs'> here</a>.

Figure 65. Clouds of models of ‘grijs’. Explore it here.

As shown in Figure 66, the first order parameters have the most singlehanded weight; FOC-POS is particularly sensitive to PPMI and SOC-POS: It makes a lower difference with PPMI:weight (less than 0.2) and highest with SOC-WIN:4 (greater than 0.4). For further inspection we’ll go to the known selection of LENGHT:FOC + SOC-WIN:4 + SOC-POS:nav, initially disregarding PPMI:selection.

Figure 66. Distances between models of 'grijs' that vary along only one parameter, colored by `PPMI` and `SOC-WIN`.

Figure 66. Distances between models of ‘grijs’ that vary along only one parameter, colored by PPMI and SOC-WIN.

FOC-POS:nav models lose two tokens in combination with FOC-WIN:5, 8 in combination with PPMI:weight and 15 with all three restrictions to the first order context together. The distance matrix shows that the loosest model is the most different, followed by its FOC-WIN:5 counterpart. The difference is less drastic when PPMI:no is replaced by PPMI:selection, but the picture is very different when PPMI:weight is replaced: there, FOC-POS makes the biggest differences.

MDS solutions look more compact in PPMI:no models than with PPMI:weight, with some sort of arms stemming from the core of the loosest models, but it is no striking difference. PPMI:weight + FOC-POS:nav models do have a small cluster of gray cars, with collocates such as “Mercedes”, “BMW” and “brommer” ‘moped’, including “kenteken” ‘license plates’ (this cluster remains as such in t-SNE models, but the loosest one excludes “grijs kenteken” ‘gray license plate’ from it). Color coding shows that the various senses do tend to stick together in certain areas, which overlap in varying degrees. While grijs_1 ‘gray-visual’ is clearly the dominant group, it does not cover more than half the cloud: it actually tends to take up a quarter of it and expand with some satellites. grijs_2 ‘cloudy’ takes better shape with PPMI:weight –very neatly with FOC-WIN:5 + FOC-POS:all–, mostly sharing its region with grijs_6 ‘gray-moral’ and in diametrical opposition to the dense clear group of grijs_3 ‘gray-hair’, in every model. grijs_5 ‘boring’ takes the quarter opposite to grijs_1 ‘gray-visual’, but spread around so that less tokens take up as much surface. grijs_4 ‘old’ and not_listed, on the other hand, are spread around all around the clouds (Figure 67).

Figure 67. Tokens of 'grijs' in the MDS solutions of the selected models.

Figure 67. Tokens of ‘grijs’ in the MDS solutions of the selected models.

The t-SNE solutions start up as uniform archipelagos with perplexity 5, more disperse with PPMI:weight, and a number of clusters start to take shape with higher perplexity, although never achieving obvious separation. For some clouds it rather looks like a big mass with a couple of denser nuclei and a small number of satellites (like one small dense cluster of foods, such as “grijze garnalen” ‘gray shrimp’ and “grijs brood” ‘brown bread, lit. gray bread’). In any case, from the beginning color coding shows that the senses tend to stick together, without too much overlap. grijs_1 ‘color-visual’ has three to five clusters with some tokens in between, distinguishing clothers, materials (e.g. stone), food and cars, with all models relatively good at separating them. grijs_2 ‘cloudy’ has its own cluster in FOC-WIN:10 models (to the point that it even stands out without color coding), which is much less dense in FOC-WIN:5, although rather decent in FOC-WIN:5 + FOC-POS:all + PPMI:weight. The grijs_3 ‘gray-hair’ cluster is also quite tight and persistant across all models and tends to stick close to the cluster of gray clothes. grijs_6 ‘gray-moral’ has two small tight clusters of “grijs gebied” ‘gray area’ and “grijze zone” ‘gray zone’ in PPMI:weight and FOC-WIN:5 + FOC-POS:nav and a single less dense area in the other three. grijs_4 ‘old’, gray_5 ‘boring’ and not_listed are spread around, across pretty much the whole cloud. Switching PPMI:no to PPMI:selection doesn’t make too much of a difference.

Figure 68. Tokens of 'grijs' in the t-SNE solutions (perplexity of 30) of the selected models.

Figure 68. Tokens of ‘grijs’ in the t-SNE solutions (perplexity of 30) of the selected models.

The second order parameters do not make much of a difference: LENGTH does, to some degree, and in that case LENGTH:FOC looks a bit better than LENGTH:5000.

A configuration that seems interesting for further inspection of “grijs” is the one with FOC-WIN:10 + FOC-POS:all + PPMI:weight for first order context and SOC-WIN:4 + SOC-POS:nav + LENGTH:FOC for second order.

heet

The adjective heet was tagged with 6 definitions, reproduced in Table 37. It basically translates to English as “hot” or “warm”; the first three definitions correspond to its application to objects that feel hot to the touch (heet_1), which covered half the cases of the pilot concordance, to bodies that feel warm themselves (heet_2) and to the weather (heet_3); then one is synesthetic, meaning “spicy” (heet_4), and two are metaphoric: one in the domain of sex (heet_5) and one in the domain of conflict (heet_6). heet_2 ‘hot-body’ and heet_4 ‘aroused’ were not attested in the pilot sample; heet_3 ‘hot-weather’ did have a decent frequency, while heet_4 ‘spicy’ and heet_6 ‘conflictive’ were quite infrequent. About half the instances of heet_1 ‘hot-objects’ were actually idiomatic expressions, where heet itself is not being used metaphorically, but the whole composite image (“hete hangijzer” ‘hot iron’, “hete aardappel” ‘hot potato’) is. The remaining 8 tokens were either a wrong part of speech (e.g. a form of the verbe heten ‘call, name’) or too ambiguous.

Table 37. Definitions of ‘heet’.
code definition example freq
heet_1 (van dingen) zeer warm een gloeiend hete kachel 20
heet_2 (van het lichaam) warm aanvoelend, een hogere temperatuur dan normaal hebbend hete wangen, het heet hebben 0
heet_3 (van het weer) zeer warm hete dagen, hete zomer 8
heet_4 (van voedsel) pikant hete sauzen 1
heet_5 (van personen) sexueel hartstochtelijk, geil een hete bok 0
heet_6 (van gebeurtenissen, periodes e.d.) gekenmerkt door heftige strijd het ging er heet aan toe, een hete herfst 3

Sense distribution

The sample consists of 360 tokens (9 batches) out of 10676 occurrences in the QLVLNewsCorpus; the distribution of the majority senses of each batch, as well as the pilot-based estimate and the overall distribution, are reproduced in Figure 69. The distributions of the annotations (not majority senses) by annotator are shown in Figure 70. Batch 4 was annotated by 4 annotators. While the distribution varies a bit across batches, with a low proportion of heet_1 ‘hot-object’ in batches 1 and 3 in favour fo geen ‘none of the above’ (probably from misunderstanding how to annotate cases such as “hete aardappel” ‘hot potato’) and other minor fluctuations, the overall distribution is extremely similar to the expected. heet_2 ‘hot-body’ and heet_5 ‘aroused’ are indeed attested, but infrequently and not in all batches. The number of tokens with geen ‘none of the above’ as majority sense or with no agreement at all is also very high.

Figure 69. Distribution of majority senses of 'heet' per batch

Figure 69. Distribution of majority senses of ‘heet’ per batch

Figure 70. Distribution of sense annotations of 'heet' per annotator, grouped by batch.

Figure 70. Distribution of sense annotations of ‘heet’ per annotator, grouped by batch.

“heet” is an adjective with a number of attested senses: one core physical one that usually covers half the instances (but includes idiomatic expressions), another one physical on a different domain with half as many instances, a metaphorical one, a bit less frequent, and then infrequent literal and derived senses. It also has a number of challenging issues.

Confusion matrix

Matrices

The confusion matrix between the majority senses and other tagged senses can be seen in Table 38 (raw number of tokens with such senses assigned) and Table 39 (mean confidence of such sense annotation in each token). Some confusion can be expected between literal senses if the objects are not clear (or prototypical) enough, or between heet_3 ‘hot-weather’ and heet_6 ‘conflictive’ for instances of ambiguity or double sense, but none other. While the annotators were instructed to annotate idiomatic expressions with the literal tags, not all of them read (or applied) the instructions, and instead assigned a geen ‘none of the above’ tag and specified that it was an idiomatic expression (that has already been attested in previous items).

Figure 71 has indeed quite a clear diagonal and a lot of 0s. heet_1 ‘hot-object’ is not only the most frequent but also the most confused, with particularly high numbers with heet_6 ‘conflictive’; the tokens with no agreement also involve mostly heet_1 ‘hot-object’ and heet_6 ‘conflictive’ annotations. The confusion between heet_3 ‘hot-weather’ and heet_6 ‘conflictive’, instead, is neglectable. heet_3 ‘hot-weather’, (infrequent) heet_4 ‘spicy’ and heet_5 ‘aroused’ have a high proportion of annotations with full agreement, as well as wrong_lemma.

Table 38. Non weighted sense matrix of ‘heet’ senses. Proportion of tokens with full agreement per sense-tag is: heet_1: 0.61, heet_3: 0.74, heet_4: 0.8, heet_5: 0.7, heet_6: 0.32, not_listed: 0.17, wrong_lemma: 0.74.
senses heet_1 heet_2 heet_3 heet_4 heet_5 heet_6 between not_listed unclear wrong_lemma
heet_1 178 3 3 8 2 28 4 19 1 2
heet_2 4 5 1 0 0 0 0 0 0 0
heet_3 5 1 65 0 1 7 0 3 0 0
heet_4 0 0 1 5 0 0 0 0 0 0
heet_5 1 0 1 0 10 1 0 0 0 0
heet_6 13 1 3 0 1 34 0 4 1 0
not_listed 3 1 0 0 0 1 0 6 0 0
unclear 0 0 0 0 0 0 0 0 1 1
wrong_lemma 0 0 0 0 0 0 0 0 6 23
no_agreement 28 15 5 1 6 25 1 11 4 2
total 232 26 79 14 20 96 5 43 13 28

heet_3 ‘hot-weather’ annotations and the agreeing annotations of heet_1 ‘hot-object, heet_4 ’spicy’ and wrong_lemma seem to have the highest mean confidences.

Table 39. Weighted sense matrix of ‘heet’ senses. Mean confidence across the lemma is 3.83; values above are darker and boldened. Median confidence across the lemma is 4.
senses heet_1 heet_2 heet_3 heet_4 heet_5 heet_6 between not_listed unclear wrong_lemma
heet_1 3.95 2.67 4 3.62 2.5 3.43 1.5 3.95 4 0.5
heet_2 3.75 3.7 5 0 0 0 0 0 0 0
heet_3 3.4 5 4.37 0 5 3.14 0 3 0 0
heet_4 0 0 4 4.4 0 0 0 0 0 0
heet_5 5 0 3 0 3.57 3 0 0 0 0
heet_6 3.08 2 2 0 2 3.31 0 2.75 0 0
not_listed 4.67 1 0 0 0 2 0 2.94 0 0
unclear 0 0 0 0 0 0 0 0 2 0
wrong_lemma 0 0 0 0 0 0 0 0 3 4.05
no_agreement 2.79 2.13 2 3 2 3.52 1 2.82 1.75 2

There were 23 tokens across seven of the nine batches where all annotators assigned geen ‘none of the above’ and the majority justified it by pointing out that the target was actually geen adjective but a form of the verb heten ‘to call, name’ (or in one case a typo that was supposed to be het ‘the’). In one of the batches with five of such cases one of the annotators only (redundantly) commented that none of the senses was appropriate, without specifying the part of speech issue, and in one other token one annotator was too confused to assume that this was the explanation. In any case, this group of tokens should be a good test for the sensibility of the models to part of speech.

Alternative senses

In 6 tokens, the majority of the annotators assigned geen ‘none of the above’ and suggested an alternative sense. Two of them were instances of “hete aardappel” ‘hot potato’, but the others were indeed senses that were not covered by the given definitions. Example (112) matches heet_2 ‘hot-body’ (and indeed one of the annotators assigned that sense), but with a metaphorical sense: not getting hot nor warm from something means that it doesn’t affect the person. (113) and (114) are examples of a sense that was looked for in the pilot concordance but not attested, meaning “popular”. (115), finally, exemplifies another fixed expression, “heet van de naald” ‘very recent(ly made)’.

  1. de Nederlandse teambaas Van Vliet de pers te woord . " Ik word niet heet of koud van al die insinuaties . We hebben weer bewezen hoe homogeen deze
    de Dutch team boss Van Vleit said to the press. "I’m not bothered by all those insinutations (lit. I don’t get hot nor cold from all those insinuations). We have demonstrated again how homogeneous these
  2. democratie wel interessanter . Kijk ook op www.stemhok.nl : daar heet hij ’ de heetste politicus van het moment ’ . MICHIEL HULSHOF , Amsterdam Door zijn
    democracy more interesting. Look also on www.stemhok.nl: there is he also called ‘the hottest politician of the moment’. MICHIEL HULSHOF, Amsterdam. With his
  3. Chart-uitzendingen , elke woensdag en vrijdag . We geven tien exemplaren weg van dit hete album . Mail vandaag , woensdag 11 juni tijdens de kantooruren naar .
    Chart-broadcasts, every Wednesday and Friday. We gave ten copies of this hot album away. Mail today, Wednesday June 11 during working hours to .
  4. het konvooi bestookte van Amerikanen en Koerdische strijders waarin Simpson reisde . Hij deed heet van de naald verslag van deze catastrofe . De tolk van de BBC ,
    bombarded the convoy of American and Kurdish fighters where Simpson travelled. He gave a very recent (lit. hot from the needle) report of this catastrophe. The interpreter of the BBC?

No agreement

There are 33 tokens (almost a tenth!) with no agreement between the annotators, and only one was a case of four annotators split between two senses. heet_1 ‘hot-object’ and heet_6 ‘conflictive’ are the most problematic; in fact, in 11 of this cases the disagreement was between these two senses and heet_2 ‘hot-body’, in 7 between them and geen ‘none of the above’, and the one case of four annotators was –wait for it– between heet_1 ‘hot-object’ and heet_6 ‘conflictive’. The first group was made of 10 occurrences (across three batches) of “hete adem van iemand in de nek” ‘someone’s hot breath in the neck’, where that “someone” is not necessarily a human being but could also be an organization, meaning that they are keeping close tabs and a lot of pressure on somebody. The eleventh case was one of “hete tranen” ‘warm tears’, likely metaphoric. The other cases represent a variety of fixed expressions or different senses; two were cases of “heet van de naald” ‘very recent(ly made)’, like in (115), applied to verkiezing ‘election’ and datering ‘dating (estimation of age)’; the rest will be described presently.

In (116) through (119), there is a composite metaphorical expression that is more or less related to a known metaphorical meaning of the target (heet_6 ‘conflictive’) but inside which the meaning is literal heet_1 ‘hot-object’. The annotations of (116) had low and medium confidence: the annotator that assigned geen ‘none of the above’ specified that it was a figurative meaning (helpful!) and the one that assigned heet_1 ‘hot-object’, that it was an expression (the requested behavior); with (117) the annotators that assigned heet_6 ‘conflictive’ and geen ‘none of the above’ were the ones to identify and even explain the expression, which means that a situation is less serious than expected. The expression in (118) was also identified and explained by the same annotators, which assigned the same sense tags as in (117) (in both cases also the one that assigned geen ‘none of the above’ included an url to a dictionary). In this case, the idiomatic expression is roughly equivalent to “beat around the bush”; an interesting detail is that the object should actually be brij ‘porridge’ instead of brei ‘knit’, but since the pronunciation is so similar it seems to be a [frequent source of confusion](https://onzetaal.nl/taaladvies/brij-brei. (119) received two annotations for heet_1 ‘hot-object’ with high confidence and one comment stating it was an expression and two annotations of *heet_6** ‘conflictive’ with medium and minimum confidence.

  1. Ze zijn bang dat de oud-dictator probeert te vluchten als de grond hem in Senegal te heet onder de voeten wordt . Het huis van Habré , vlakbij het internationale vliegveld
    They fear that the old-dictator will try to flee once the ground becomes too hot under his feet. The house of Habré, close to the international airport
  2. graven . " Ton Hooijmaijers van de VVD denkt ook dat de soep minder heet wordt gegeten dan ze nu wordt opgediend . Als het rijk wat meer geld
    dig. "Tom Hooijmaijers from the VVD also thinks that the situation is not as serious as expected (lit. the soup is eaten less hot than it is now served). If the rich […] some more money
  3. Daar houd ik wel van . " Een type dat om de hete brei heendraait , is Joke inderdaad niet . De geblokte chauffeuse , die zich
    I do like that. "A guy that beats around the bush (lit. circles around the hot porridge, Joke is certainly not. The blocked?? chauffeur, who
  4. moeten zien . Vooral het functioneren van een collectie binnen zo’n museum bleek een heet hangijzer . Zo hield Hendrik Driessen , directeur van het privé-museum De Pont in
    must seen. Mostly the functioning of a collection inside such a museum turned out to be a big problem (lit. hot from the needle) iron). So said Hendrik Driessen, director of the De Pont private museum in

In (120) and (121) the object of which the target is being predicated is no physical object that could become hot or warm (“festivalnieuws” ‘news of the festival’ and “nieuwsfeiten” ‘news facts’ respectively): we are dealing with the meaning of “popular” in the later, like in (113) and (114), and probably heet_6 ‘conflictive’ in the latter. The annotations here were not particularly helpful: for (120), the annotator that assigned heet_1 ‘hot-object’ did add that it was a figurative meaning, but the one that assigned geen ‘none of the above’ just commented “heet, broeierig” ‘hot, scalding’; for (121), only the annotator that selected geen ‘none of the above’ made a comment (which was compulsory), in which they stated that they didn’t know whether “nieuwsfeiten” ‘news facts’ was meant figuratively.

  1. l Festivalitis Jimtv 18.30 en 22.00 Het heetste festivalnieuws , de beste interviews en exclusieve concertbeelden . Met officieel festivalexpert Ilse Liebens
    Festivalitis Jimtv 18.30 and 22.00 The hottest festival news, the best interviews and exclusive concerti pictures. With the official festival expert Ilse Liebens
  2. . Of als een Emile Ratelband die zijn discipelen op blote voeten door gloeiend hete nieuwsfeiten laat rennen , onder het uitroepen van de kreet ’ kopij ! kopij ! kopij
    . Or if an Emile Ratelband that lets his disciples run barefoot through scalding hot news facts, under the cry of ’copy! copy! copy

The rest of the cases with no agreement whether either similar to some of the expressions already described, really ambiguous cases or nonsensical annotations.

Nephology of heet

A first impression on the clouds relates to the stress values of the dimensionality reduction and the parameters that make the strongest distinctions between models. We have 144 models of heet created on 25/03/2020, modeling between 348 and 359 tokens. The stress value of the MDS solution for the cloud of models is 0.161.

The main split in the cloud of models is made by FOC-WIN along the vertical axis; each half is made of a small cloud on the left with PPMI:weight models, split orthogonally by FOC-POS and SOC-WIN, and middle section of FOC-POS:nav and a right, quite spread section of FOC-POS:all; these two split by the other two PPMI:value and with some SOC-WIN groups (Figure 72).

Figure 72. Clouds of models of 'heet'. Explore it <a href='https://montesmariana.github.io/NephoVis/level1.html?type=heet'> here</a>.

Figure 72. Clouds of models of ‘heet’. Explore it here.

As Figure 73, FOC-POS, FOC-WIN and PPMI are the only parameters that singlehandedly make a difference above 0.2, and that difference is greater with SOC-WIN:4 than with SOC-WIN:10. FOC-POS:all also means larger differences between models that vary only along FOC-WIN, LENGTH or SOC-WIN, although to a lesser degree than PPMI. To compare the actual models that vary along these parameters, a selection with LENGTH:FOC + SOC-WIN:10 + SOC-POS:nav will be examined, initially discarding PPMI:selection models.

Figure 73. Distances between models of 'heet' that vary along only one parameter, colored by `PPMI` and `SOC-WIN`.

Figure 73. Distances between models of ‘heet’ that vary along only one parameter, colored by PPMI and SOC-WIN.

One token is lost by FOC-POS:nav models, 7 more in combination with PPMI:weight and 3 more with all three restrictions in place. The distance matrix shows relatively low values, but the loosest model stands out with higher differences, as does, to a lesser degree, its FOC-WIN:5 counterpart. The values seem a bit lower when PPMI:no is replaced by PPMI:selection; when PPMI:weight is replaced, instead, the largest differences lie between the FOC-WIN:10 + FOC-POS:nav models on one side and the FOC-WIN:5 + FOC-POS:all on the other.

The MDS solutions give more compact models with PPMI:no and a big dense cluster with a more disperse area and one or two smaller clusters (for “hete hangijzer” ‘hot iron’ and “hete adem” ‘hot breath’) in the PPMI:weight models. Color coding shows a relatively good separation of senses: heet_1 ‘hot-object’ is clearly dominant, but also densely packed in about a quadrant and the models that do identify the “hete hangijzer” ‘hot iron’ cluster pull it well apart from it; heet_3 ‘hot-weather’ also has its own dense region which avoids the big heet_1 ‘hot-object’ area quite well in models that have at least two of the first order restrictions (PPMI:weight, FOC-WIN:5 and FOC-POS:all). The rest of the senses and the heten ‘call, name’ cases are spread around the cloud without specific areas other than ‘not with the main clouds’, although that is not so strict either.

The t-SNE solutions have an archipelago structure with perplexity 5 and two or three small clusters (at least “hete hangijzer” ‘hot iron’ and “hete adem” ‘hot breath’) around a larger mass with some nuclei with perplexity 20; with larger perplexities, the clusters grow tighter and farther apart and the central mass more dispense and uniform. Color coding shows that the larger subcloud is mainly split between heet_1 ‘hot-object’ (which has some tighter pockets of “hete aardappel” ‘hot potato’ close to but separated from one with actual hot foods) and heet_3 ‘hot-weather’ tokens. There are no other evident agglomerations of tokens of the same sense (Figure 74).

Figure 74. Tokens of 'heet' in the t-SNE solutions (perplexity 30) of the selected models

Figure 74. Tokens of ‘heet’ in the t-SNE solutions (perplexity 30) of the selected models

PPMI:selection models resemble PPMI:weight models more than their PPMI:no counterparts. Moreover, the grouping seems a bit more structured and balanced with FOC-WIN:10 + FOC-POS:nav, and once these parameters are set, the second order parameters make a neglectable difference.

For further inspection of “heet” it would be interesting to look into clouds with FOC-WIN:10 + FOC-POS:nav + PPMI:weight | PPMI:selection as first order parameters and SOC-WIN:4 + SOC-POS:nav + LENGTH:FOC for the second order.

References

De Pascale, S. (2019). Token-based vector space models as semantic control in lexical lectometry (PhD thesis). Retrieved from https://lirias.kuleuven.be/retrieve/549451

  1. The outliers keep the same first order context words in PPMI:selection and PPMI:no, but are much further from the main cloud in the first case. It might be the case that the rest of the tokens are very different or that the selection of second order context words is very different, because of the LENGTH:FOC parameter (but the picture is very similar with FOC:5000.↩

  2. The comment read: “Ik twijfel tussen betekenis 1 en 3, maar ik denk dan toch eerder 1 (in figuurlijke betekenis: een lezing met scherpe kanten), Eerste betekenis, maar in de metaforische zin - zonder empathie voor de luisteraars, ik twijfel tussen 2 en 3”.↩

  3. The clear split in the LENGTH column is made by the models with LENGTH:5000 | LENGTH:10000 (at the bottom), against those with other alternations.↩

  4. The original comment reads: “Geen van de betekenissen past hier echt. Dof betekent hier inderdaad eerder ‘zonder glans’ maar dan in de context van ‘zonder de glans van een beroemd, bekend leven’. Het is dus niet letterlijk zonder een zichtbare glans (zoals in de eerste betekenislijn), het is ook niet echt waarschijnlijk dat Halliday verlangt naar een ‘niet opgewekt, lusteloos’ leven.De andere twee betekenissen zijn zeker niet van toepassing aangezien het hier niet om geluid gaat en het heel vreemd zou zijn om te verlangen naar een ‘niet scherp voor de geest staand’ leven. Dit voorbeeld komt nog twee keer terug.”.↩

  5. (Probably the annotators tagged the de ‘the’ closest to the target, but a known bug in the annotation tool misregisterd it and they didn’t correct it when warned.)↩

  6. (The " was actually rendered as andquot;, from a known bug)↩

  7. One of the annotators did comment “foute woordsoort” ‘wrong part of speech’ but then expanded suggesting the sense of “zwaar” ‘tough’ and remarking it was figurative, for which I suspect they don’t know what “foute woordsoort” really means.↩